You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov> on 2018/10/10 18:34:28 UTC
NiFi fails on cluster nodes
Hello,
We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here<https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
at org.apache.nifi.NiFi.<init>(NiFi.java:102)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I'm not sure where to look for clues.
Thanks in advance,
Alexander
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
Also, not sure if this provides anything additional to what has
already been mentioned on this thread, but this morning I wrote up the
exact steps I followed to create a secure 2 node cluster to test the
1.8.0 release candidate.
https://bryanbende.com/development/2018/10/23/apache-nifi-secure-cluster-setup
On Tue, Oct 23, 2018 at 2:43 PM Bryan Bende <bb...@gmail.com> wrote:
>
> So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> It doesn't really make sense how the second node would form its own cluster.
> On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C]
> <al...@nih.gov> wrote:
> >
> > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
> >
> >
> >
> > 2018-10-23 13:44:43,628 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Client attempting to establish new session at /<host-2 IP address>:50412
> >
> > 2018-10-23 13:44:43,629 INFO [SyncThread:0] o.a.zookeeper.server.ZooKeeperServer Established session 0x166a1d139590002 with negotiated timeout 4000 for client /<host-2 IP address>:50412
> >
> >
> >
> > I apologize for bugging you with all this, converting our standalone NiFi instances into cluster nodes turned out to be much more challenging than we had anticipated…
> >
> >
> >
> > -----Original Message-----
> > From: Bryan Bende <bb...@gmail.com>
> > Sent: Tuesday, October 23, 2018 1:17 PM
> > To: users@nifi.apache.org
> > Subject: Re: NiFi fails on cluster nodes
> >
> >
> >
> > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
> >
> > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
> >
> > >
> >
> > >
> >
> > >
> >
> > > -----Original Message-----
> >
> > > From: Bryan Bende <bb...@gmail.com>
> >
> > > Sent: Tuesday, October 23, 2018 12:36 PM
> >
> > > To: users@nifi.apache.org
> >
> > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > >
> >
> > >
> >
> > > That means the user representing host-1 does not have permissions to proxy.
> >
> > >
> >
> > >
> >
> > >
> >
> > > You can look in authorizations.xml on nifi-1 for a policy like:
> >
> > >
> >
> > >
> >
> > >
> >
> > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
> >
> > >
> >
> > > resource="/proxy" action="W">
> >
> > >
> >
> > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
> >
> > >
> >
> > > </policy>
> >
> > >
> >
> > >
> >
> > >
> >
> > > That user identifier should point to a user in users.xml like:
> >
> > >
> >
> > >
> >
> > >
> >
> > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
> >
> > >
> >
> > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
> >
> > >
> >
> > > Government, C=US"/>
> >
> > >
> >
> > >
> >
> > >
> >
> > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
> >
> > >
> >
> > >
> >
> > >
> >
> > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Hi Bryan,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · restarted one instance in the standalone mode
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · granted them the “proxy user requests” privileges
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · restarted the node on host-1
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Insufficient Permissions
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > The nifi-user.log also contains
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
> >
> > >
> >
> > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
> >
> > >
> >
> > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS,
> >
> > >
> >
> > > > O=U.S. Government, C=US
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > From your experience, what the most likely causes for this exception?
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Thank you,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Alexander
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > -----Original Message-----
> >
> > >
> >
> > > > From: Bryan Bende <bb...@gmail.com>
> >
> > >
> >
> > > > Sent: Monday, October 22, 2018 1:25 PM
> >
> > >
> >
> > > > To: users@nifi.apache.org
> >
> > >
> >
> > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Yes, to further clarify what I meant...
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > - In standalone mode, use the UI to add users for the DN's of the
> >
> > >
> >
> > > > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2,
> >
> > > > OU=NIFI)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > - In the UI, grant those users Write access to "Proxy"
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > - Convert to a cluster and keep your same authorizers.xml,
> >
> > > > users.xml,
> >
> > >
> >
> > > > and authorizations.xml when you setup your cluster, this way all
> >
> > > > your
> >
> > >
> >
> > > > users and policies are already setup and the Initial Admin and Node
> >
> > >
> >
> > > > Identities are not needed
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > -----Original Message-----
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > From: Bryan Bende <bb...@gmail.com>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Sent: Monday, October 22, 2018 12:48 PM
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > To: users@nifi.apache.org
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > This blog post shows how to setup a secure 2 node cluster:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-au
> >
> > > > > th
> >
> > >
> >
> > > > > or
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > ization-and-multi-tenancy
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <authorizer>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <identifier>file-provider</identifier>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Authorizations
> >
> > >
> >
> > > > > File">./conf/authorizations.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Users File">./conf/users.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial Admin Identity">CN=bbende,
> >
> > >
> >
> > > > > OU=ApacheNiFi</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Legacy Authorized Users File"></property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Node Identity 1">CN=localhost,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > </authorizer>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > You need to add the the users to the user-group-provider and then to the access-policy-provider...
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <userGroupProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <identifier>file-user-group-provider</identifier>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Users File">./conf/users.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Legacy Authorized Users File"></property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial User Identity 1">CN=bbende,
> >
> > >
> >
> > > > > OU=Apache NiFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial User Identity 2">CN=nifi-host-1,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial User Identity 2">CN=nifi-host-2,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > </userGroupProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <accessPolicyProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <identifier>file-access-policy-provider</identifier>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</cla
> >
> > > > > ss
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="User Group
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Provider">composite-configurable-user-group-provider</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Authorizations
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > File">./conf/authorizations.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial Admin Identity">CN=bbende,
> >
> > > > > OU=Apache
> >
> > >
> >
> > > > > NiFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Legacy Authorized Users File"></property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Node Identity 1">CN=nifi-host-1,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Node Identity 1">CN=nifi-host-2,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > </accessPolicyProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Hi Bryan,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Are you saying that we have to run ZooKeeper on both nodes? BTW,
> >
> > >
> >
> > > > > > do
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > we still need
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > nifi.login.identity.provider.configuration.file=./conf/login-ide
> >
> > > > > > nt
> >
> > >
> >
> > > > > > it
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > y-
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > providers.xml
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Thank you,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Alexander
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > -----Original Message-----
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > From: Bryan Bende <bb...@gmail.com>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Sent: Monday, October 22, 2018 11:55 AM
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > To: users@nifi.apache.org
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > server.1=nifi-node1-hostname:2888:3888
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > server.2=nifi-node2-hostname:2888:3888
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > I wonder if anyone has run into the same problem when trying
> >
> > > > > > > to
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > configure composite authentication/authorization (LDAP and
> >
> > >
> >
> > > > > > > local
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > file)? When we use the “stand-alone” authorizers.xml file with
> >
> > >
> >
> > > > > > > the
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > addition of two extra properties
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > <property name="Node Identity 1">…
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > <property name="Node Identity 2">…
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > and let ZooKeeper start on one on the nodes, we end up with
> >
> > > > > > > two
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > one-node clusters, since apparently, the NiFi instances don’t
> >
> > >
> >
> > > > > > > talk
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > to
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > each other, but at least, they come alive…
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > From: Saip, Alexander (NIH/CC/BTRIS) [C]
> >
> > >
> >
> > > > > > > <al...@nih.gov>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Sent: Friday, October 19, 2018 11:18 AM
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > To: users@nifi.apache.org
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Subject: RE: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 2018-10-19 08:09:26,992 ERROR [main]
> >
> > >
> >
> > > > > > > o.s.web.context.ContextLoader
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Context initialization failed
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Error creating bean with name
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 'org.springframework.security.config.annotation.web.configuration.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > We
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > bS
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ecurityConfiguration': Unsatisfied dependency expressed
> >
> > > > > > > through
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > method
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > exception
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > is org.springframework.beans.factory.BeanExpressionException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Expression parsing failed; nested exception is
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Error creating bean with name
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
> >
> > >
> >
> > > > > > > Unsatisfied
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > dependency expressed through method 'setJwtAuthenticationProvider'
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > parameter 0; nested exception is
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > creating bean with name 'jwtAuthenticationProvider' defined in
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > class
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > path resource [nifi-web-security-context.xml]: Cannot resolve
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > reference to bean 'authorizer' while setting constructor
> >
> > >
> >
> > > > > > > argument;
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nested exception is
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > creating bean with name 'authorizer': FactoryBean threw
> >
> > >
> >
> > > > > > > exception
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > on
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > object creation; nested exception is java.lang.NullPointerException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Name is null
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotati
> >
> > > > > > > on
> >
> > >
> >
> > > > > > > Be
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > an
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Po
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationB
> >
> > > > > > > ea
> >
> > >
> >
> > > > > > > nP
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > os
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > tP
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > rocessor.java:667)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.annotation.InjectionMetadata
> >
> > > > > > > .i
> >
> > >
> >
> > > > > > > nj
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ec
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > t(
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > InjectionMetadata.java:88)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotati
> >
> > > > > > > on
> >
> > >
> >
> > > > > > > Be
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > an
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Po
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanP
> >
> > > > > > > os
> >
> > >
> >
> > > > > > > tP
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ro
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ce
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ssor.java:366)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractAutowireCapa
> >
> > > > > > > bl
> >
> > >
> >
> > > > > > > eB
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ea
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nF
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:12
> >
> > > > > > > 64
> >
> > >
> >
> > > > > > > )
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractAutowireCapa
> >
> > > > > > > bl
> >
> > >
> >
> > > > > > > eB
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ea
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nF
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:55
> >
> > > > > > > 3)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractAutowireCapa
> >
> > > > > > > bl
> >
> > >
> >
> > > > > > > eB
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ea
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nF
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.
> >
> > >
> >
> > > > > > > ge
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > tO
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > bj
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ect(AbstractBeanFactory.java:306)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.DefaultSingletonBean
> >
> > > > > > > Re
> >
> > >
> >
> > > > > > > gi
> >
> > >
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Please disregard my previous message. After I nuked again the …/state/zookeeper/version-2 directory on host-1, and restarted NiFi on both servers, we got the second instance finally joining the cluster and picking up the dataflows from the first one. Thanks everyone who helped us through, we learned quite a bit along the way!
From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
Sent: Thursday, October 25, 2018 7:48 AM
To: users@nifi.apache.org
Subject: RE: NiFi fails on cluster nodes
Attaching the nifi.properties files for both hosts (the FQDNs are replaced with <host-1> and <host-2>, respectively, and the passwords with <redacted>). Frankly, I’m puzzled myself as to where 0.0.0.0 comes from. I restarted both NiFi instances this morning, and here is some excerpts from the host-2 nifi-app.log file:
2018-10-25 07:07:42,007 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected...
2018-10-25 07:07:42,053 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
2018-10-25 07:07:42,117 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2018-10-25 07:07:42,139 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2018-10-25 07:07:42,142 INFO [main] o.apache.nifi.controller.FlowController The Election for Cluster Coordinator has already begun (Leader is 0.0.0.0:11443). Will not register to be elected for this role until after connecting to the cluster and inheriting the cluster's flow.
2018-10-25 07:07:42,143 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election.
2018-10-25 07:07:42,143 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
2018-10-25 07:07:42,145 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election.
2018-10-25 07:07:42,146 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started
2018-10-25 07:07:42,146 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started
2018-10-25 07:07:42,147 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2018-10-25 07:07:44,666 WARN [main] org.glassfish.jersey.internal.Errors The following warnings have been detected: WARNING: The (sub)resource method getCounters in org.apache.nifi.web.api.CountersResource contains empty path annotation.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.nifi.web.api.DataTransferResource.transferFlowFiles(java.lang.String,java.lang.String,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,javax.servlet.ServletContext,java.io.InputStream), should not consume any entity.
WARNING: The (sub)resource method submitProvenanceRequest in org.apache.nifi.web.api.ProvenanceResource contains empty path annotation.
WARNING: The (sub)resource method getAccessStatus in org.apache.nifi.web.api.AccessResource contains empty path annotation.
<...>
2018-10-25 07:07:45,248 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...
2018-10-25 07:07:45,252 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 11443
2018-10-25 07:07:45,310 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow
2018-10-25 07:07:45,325 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: <host-2>:8008
2018-10-25 07:07:45,327 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 0.0.0.0:11443; will use this address for sending heartbeat messages
2018-10-25 07:07:45,506 INFO [Process Cluster Protocol Request-1] o.a.n.c.c.flow.PopularVoteFlowElection Vote cast by <host-2>:8008; this flow now has 1 votes
2018-10-25 07:07:45,506 INFO [Process Cluster Protocol Request-1] o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from <host-2>:8008; responding with Flow Election In Progress message
2018-10-25 07:07:45,508 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 9ba2cfee-4720-43d4-bafd-6b7e81fc6557 (type=CONNECTION_REQUEST, length=14838 bytes) from <host-2>:8008 in 174 millis
2018-10-25 07:07:45,511 INFO [main] o.a.nifi.controller.StandardFlowService Requested by cluster coordinator to retry connection in 5 seconds with explanation: Cluster is still voting on which Flow is the correct flow for the cluster. Election will complete in 299 seconds
2018-10-25 07:07:47,247 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 12e3cd88-ae73-4db5-8ea2-cdf1bd5d1082 (type=NODE_CONNECTION_STATUS_REQUEST, length=97 bytes) from <host-2> in 93 millis
<...>
2018-10-25 07:12:48,089 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 0.0.0.0:11443; will use this address for sending heartbeat messages
2018-10-25 07:12:48,222 INFO [Process Cluster Protocol Request-10] o.a.n.c.c.flow.PopularVoteFlowElection Election is complete because the maximum allowed time has elapsed. The elected dataflow is held by the following nodes: [<host-2>:8008]
2018-10-25 07:12:48,223 INFO [Process Cluster Protocol Request-10] o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from <host-2>:8008; responding with DataFlow that was elected
2018-10-25 07:12:48,224 INFO [Process Cluster Protocol Request-10] o.a.n.c.c.node.NodeClusterCoordinator Status of <host-2>:8008 changed from NodeConnectionStatus[nodeId=<host-2>:8008, state=DISCONNECTED, Disconnect Code=Has Not Yet Connected to Cluster, Disconnect Reason=Has Not Yet Connected to Cluster, updateId=1] to NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=61]
2018-10-25 07:12:52,568 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 5e74a3ea-0b43-4854-b7c4-3785c2324732 (type=NODE_CONNECTION_STATUS_REQUEST, length=97 bytes) from <host-2> in 84 millis
2018-10-25 07:12:53,179 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from <host-2>/165.112.168.147:11443 due to: java.net.SocketTimeoutException: Read timed out
2018-10-25 07:12:53,185 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 0.0.0.0:11443; will use this address for sending heartbeat messages
2018-10-25 07:12:53,310 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from <host-2>:8008; responding with DataFlow that was elected
2018-10-25 07:12:53,310 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Status of <host-2>:8008 changed from NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=61] to NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=63]
2018-10-25 07:12:53,313 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request c2238153-7410-4323-8d1e-23996564d8c0 (type=CONNECTION_REQUEST, length=14838 bytes) from <host-2>:8008 in 126 millis
2018-10-25 07:12:53,315 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {<host-2>:8008=NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=63]} to {<host-2>:8008=NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=63]}
2018-10-25 07:12:53,332 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow
2018-10-25 07:12:53,339 INFO [main] o.a.nifi.controller.StandardFlowService Setting Flow Controller's Node ID: <host-2>:8008
2018-10-25 07:12:53,340 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator This node is now connected to the cluster. Will no longer require election of DataFlow.
2018-10-25 07:12:53,341 INFO [main] o.apache.nifi.controller.FlowController Cluster State changed from Not Clustered to Clustered
What is especially weird is that there is no mention of host-1 whatsoever in this log, even though ZooKeeper is only running there.
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>>
Sent: Wednesday, October 24, 2018 3:48 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
The part of the logs that might be suspicious is where it says "Cluster Coordinator is located at 0.0.0.0:11443".
I'm not totally sure, but I don't think it should have 0.0.0.0 for the address there.
What do you have set for nifi.cluster.node.address on each node? If it is not set can try setting it to the appropriate hostname for each cluster?
I'd be interested to see your nifi.properties for each node if you are able/willing to share that somehow.
On Wed, Oct 24, 2018 at 3:09 PM Viking K <cy...@hotmail.com>> wrote:
>
> Hi
> Don't know if this helps, but I run a 2 node cluster with 2 zookeepers.
> Did you do the zookeper myid assignment?
> Also have you set the flow election?
>
> I do it like this for my 2 instances,
>
> mkdir -p /nifi/state/zookeper/instance-1;echo 1
> >/nifi/state/zookeper/instance-1/myid
> mkdir -p /nifi/state/zookeper/instance-2;echo 2
> >/nifi/state/zookeper/instance-2/myid
>
> Then the cluster conf looks like this, its written in YML since I got a wrapper but its pretty self explaining.
>
> test:
> instances:
> host1:
> nifi.properties:
> nifi.cluster.node.address: 'host1.test.com'
> zookeeper.properties:
> clientPort: '2181'
> dataDir: '/nifi/state/zookeper/instance-1'
> host2:
> nifi.properties:
> nifi.cluster.node.address: 'host2.test.com'
>
> zookeeper.properties:
> clientPort: '2182'
> dataDir: '/nifi/state/zookeper/instance-2'
>
> common_configuration:
> nifi.properties:
> nifi.cluster.is.node: 'true'
> nifi.cluster.flow.election.max.candidates: '2'
> nifi.zookeeper.connect.string: 'host1.test.com:2181,host2.test.com:2182'
> nifi.state.management.embedded.zookeeper.start: 'true'
>
> bootstrap.conf:
> java.arg.2: '-Xms10g'
> java.arg.3: '-Xmx10g'
> run.as: 'nifi'
> zookeeper.properties:
> server.1: 'host1.test.com:2888:3888'
> server.2: 'host2.test.com:2889:3889'
>
> /Viking
> ________________________________
> From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>>
> Sent: Wednesday, October 24, 2018 5:47 PM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: RE: NiFi fails on cluster nodes
>
>
> I did what you suggested. There aren’t any errors in the log, although here is a warning:
>
>
>
> 2018-10-24 13:34:12,042 WARN [main]
> o.a.nifi.controller.StandardFlowService Failed to connect to cluster
> due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed
> unmarshalling 'CONNECTION_RESPONSE' protocol message from
> <host-2>/<host-2 IP address>:11443 due to:
> java.net.SocketTimeoutException: Read timed out
>
> 2018-10-24 13:34:12,049 INFO [main]
> o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster
> Coordinator is located at 0.0.0.0:11443; will use this address for
> sending heartbeat messages
>
> 2018-10-24 13:34:12,174 INFO [Process Cluster Protocol Request-2]
> o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from
> <host-2>:8008; responding with DataFlow that was elected
>
> 2018-10-24 13:34:12,175 INFO [Process Cluster Protocol Request-2]
> o.a.n.c.c.node.NodeClusterCoordinator Status of <host:8008 changed
> from NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING,
> updateId=61] to NodeConnectionStatus[nodeId=<host-2>:8008,
> state=CONNECTING, updateId=63]
>
>
>
> Please let me know if you want to see other cluster related INFO type log messages.
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Wednesday, October 24, 2018 12:08 PM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Is there anything interesting (errors/warnings) in nifi-app.log on host 2 during start up?
>
>
>
> Also, I'm not sure if this will do anything different, but you could try clearing the ZK state dir to make sure all the info in ZK is starting fresh...
>
>
>
> - Shutdown both nodes
>
> - Remove the directory nifi/state/zookeeper/version-2 on host 1 (not
> the whole ZK dir, just version-2)
>
> - Start nifi 1 and wait for it be up and running
>
> - Start nifi 2
>
>
>
> On Wed, Oct 24, 2018 at 11:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > Yes, that setting is the same for on both hosts. I attach the UI screenshots taken on those. Please note that host’s FQDNs have been removed.
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>>
>
> > Sent: Wednesday, October 24, 2018 9:25 AM
>
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > Many services can share a single ZooKeeper by segmenting their data under a specific root node.
>
> >
>
> >
>
> >
>
> > The root node is specified by nifi.zookeeper.root.node=/nifi so if those were different on each node then it would form separate clusters.
>
> >
>
> >
>
> >
>
> > Can you show screenshots of the cluster information from each node?
>
> >
>
> >
>
> >
>
> > May need to upload them somewhere and provide links here since attachments don't always make it through.
>
> >
>
> > On Wed, Oct 24, 2018 at 8:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > The ZooKeeper related settings in the nifi.properties files on both hosts are identical, with the exception of nifi.state.management.embedded.zookeeper.start, which is ‘true’ on host-1 and ‘false’ on host-2. Moreover, if I shut down NiFi on host-1, it crashes on host-2. Here is the message in the browser window:
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Action cannot be performed because there is currently no Cluster Coordinator elected. The request should be tried again after a moment, after a Cluster Coordinator has been automatically elected.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > I even went as far as commenting out the server.1 line in the zookeeper.properties file on host-1 before restarting both NiFi instances, which didn’t change the outcome.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > When I look at the NiFi Cluster information in the UI on host-1, it shows the status of the node “CONNECTED, PRIMARY, COORDINATOR”, whereas on host-2 just “CONNECTED”. I don’t know if this tells you anything.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > BTW, what does “a different location in the same ZK” mean?
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > Sent: Tuesday, October 23, 2018 3:02 PM
>
> >
>
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > The only way I could see that happening is if the ZK config on the second node pointed at a different ZK, or at a different location in the same ZK.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > For example, if node 1 had:
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.connect.string=node-1:2181
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.connect.timeout=3 secs
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.session.timeout=3 secs
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.root.node=/nifi
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Then node 2 should have exactly the same thing.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > If node 2 specified a different connect string, or a different root node, then it wouldn't know about the other node.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > That's exactly the case.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > >
>
> >
>
> > > > Sent: Tuesday, October 23, 2018 2:44 PM
>
> >
>
> > >
>
> >
>
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > >
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > It doesn't really make sense how the second node would form its own cluster.
>
> >
>
> > >
>
> >
>
> > > > On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > 2018-10-23 13:44:43,628 INFO
>
> >
>
> > >
>
> >
>
> > > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
>
> >
>
> > >
>
> >
>
> > > > > o.a.zookeeper.server.ZooKeeperServer Client attempting to
>
> >
>
> > > > > establish
>
> >
>
> > >
>
> >
>
> > > > > new session at /<host-2 IP address>:50412
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
>
> >
>
> > >
>
> >
>
> > > > > o.a.zookeeper.server.ZooKeeperServer Established session
>
> >
>
> > >
>
> >
>
> > > > > 0x166a1d139590002 with negotiated timeout 4000 for client
>
> > > > > /<host-2
>
> >
>
> > >
>
> >
>
> > > > > IP
>
> >
>
> > >
>
> >
>
> > > > > address>:50412
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > I apologize for bugging you with all this, converting our
>
> >
>
> > > > > standalone
>
> >
>
> > >
>
> >
>
> > > > > NiFi instances into cluster nodes turned out to be much more
>
> >
>
> > >
>
> >
>
> > > > > challenging than we had anticipated…
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > >
>
> >
>
> > > > > Sent: Tuesday, October 23, 2018 1:17 PM
>
> >
>
> > >
>
> >
>
> > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > >
>
> >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Sent: Tuesday, October 23, 2018 12:36 PM
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > That means the user representing host-1 does not have permissions to proxy.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > You can look in authorizations.xml on nifi-1 for a policy like:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > resource="/proxy" action="W">
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <user
>
> >
>
> > >
>
> >
>
> > > > > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > </policy>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > That user identifier should point to a user in users.xml like:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Government, C=US"/>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Hi Bryan,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · restarted one instance in the standalone mode
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · granted them the “proxy user requests” privileges
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · restarted the node on host-1
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Insufficient Permissions
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > The nifi-user.log also contains
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH,
>
> >
>
> > >
>
> >
>
> > > > > > > OU=HHS,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > O=U.S. Government, C=US
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > From your experience, what the most likely causes for this exception?
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Thank you,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Alexander
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Sent: Monday, October 22, 2018 1:25 PM
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Yes, to further clarify what I meant...
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Attaching the nifi.properties files for both hosts (the FQDNs are replaced with <host-1> and <host-2>, respectively, and the passwords with <redacted>). Frankly, I’m puzzled myself as to where 0.0.0.0 comes from. I restarted both NiFi instances this morning, and here is some excerpts from the host-2 nifi-app.log file:
2018-10-25 07:07:42,007 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected...
2018-10-25 07:07:42,053 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
2018-10-25 07:07:42,117 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2018-10-25 07:07:42,139 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2018-10-25 07:07:42,142 INFO [main] o.apache.nifi.controller.FlowController The Election for Cluster Coordinator has already begun (Leader is 0.0.0.0:11443). Will not register to be elected for this role until after connecting to the cluster and inheriting the cluster's flow.
2018-10-25 07:07:42,143 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election.
2018-10-25 07:07:42,143 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting
2018-10-25 07:07:42,145 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is a silent observer in the election.
2018-10-25 07:07:42,146 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started
2018-10-25 07:07:42,146 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started
2018-10-25 07:07:42,147 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2018-10-25 07:07:44,666 WARN [main] org.glassfish.jersey.internal.Errors The following warnings have been detected: WARNING: The (sub)resource method getCounters in org.apache.nifi.web.api.CountersResource contains empty path annotation.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.nifi.web.api.DataTransferResource.transferFlowFiles(java.lang.String,java.lang.String,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,javax.servlet.ServletContext,java.io.InputStream), should not consume any entity.
WARNING: The (sub)resource method submitProvenanceRequest in org.apache.nifi.web.api.ProvenanceResource contains empty path annotation.
WARNING: The (sub)resource method getAccessStatus in org.apache.nifi.web.api.AccessResource contains empty path annotation.
<...>
2018-10-25 07:07:45,248 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow...
2018-10-25 07:07:45,252 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 11443
2018-10-25 07:07:45,310 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow
2018-10-25 07:07:45,325 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: <host-2>:8008
2018-10-25 07:07:45,327 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 0.0.0.0:11443; will use this address for sending heartbeat messages
2018-10-25 07:07:45,506 INFO [Process Cluster Protocol Request-1] o.a.n.c.c.flow.PopularVoteFlowElection Vote cast by <host-2>:8008; this flow now has 1 votes
2018-10-25 07:07:45,506 INFO [Process Cluster Protocol Request-1] o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from <host-2>:8008; responding with Flow Election In Progress message
2018-10-25 07:07:45,508 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 9ba2cfee-4720-43d4-bafd-6b7e81fc6557 (type=CONNECTION_REQUEST, length=14838 bytes) from <host-2>:8008 in 174 millis
2018-10-25 07:07:45,511 INFO [main] o.a.nifi.controller.StandardFlowService Requested by cluster coordinator to retry connection in 5 seconds with explanation: Cluster is still voting on which Flow is the correct flow for the cluster. Election will complete in 299 seconds
2018-10-25 07:07:47,247 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 12e3cd88-ae73-4db5-8ea2-cdf1bd5d1082 (type=NODE_CONNECTION_STATUS_REQUEST, length=97 bytes) from <host-2> in 93 millis
<...>
2018-10-25 07:12:48,089 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 0.0.0.0:11443; will use this address for sending heartbeat messages
2018-10-25 07:12:48,222 INFO [Process Cluster Protocol Request-10] o.a.n.c.c.flow.PopularVoteFlowElection Election is complete because the maximum allowed time has elapsed. The elected dataflow is held by the following nodes: [<host-2>:8008]
2018-10-25 07:12:48,223 INFO [Process Cluster Protocol Request-10] o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from <host-2>:8008; responding with DataFlow that was elected
2018-10-25 07:12:48,224 INFO [Process Cluster Protocol Request-10] o.a.n.c.c.node.NodeClusterCoordinator Status of <host-2>:8008 changed from NodeConnectionStatus[nodeId=<host-2>:8008, state=DISCONNECTED, Disconnect Code=Has Not Yet Connected to Cluster, Disconnect Reason=Has Not Yet Connected to Cluster, updateId=1] to NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=61]
2018-10-25 07:12:52,568 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 5e74a3ea-0b43-4854-b7c4-3785c2324732 (type=NODE_CONNECTION_STATUS_REQUEST, length=97 bytes) from <host-2> in 84 millis
2018-10-25 07:12:53,179 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from <host-2>/165.112.168.147:11443 due to: java.net.SocketTimeoutException: Read timed out
2018-10-25 07:12:53,185 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 0.0.0.0:11443; will use this address for sending heartbeat messages
2018-10-25 07:12:53,310 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from <host-2>:8008; responding with DataFlow that was elected
2018-10-25 07:12:53,310 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Status of <host-2>:8008 changed from NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=61] to NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=63]
2018-10-25 07:12:53,313 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request c2238153-7410-4323-8d1e-23996564d8c0 (type=CONNECTION_REQUEST, length=14838 bytes) from <host-2>:8008 in 126 millis
2018-10-25 07:12:53,315 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {<host-2>:8008=NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=63]} to {<host-2>:8008=NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=63]}
2018-10-25 07:12:53,332 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow
2018-10-25 07:12:53,339 INFO [main] o.a.nifi.controller.StandardFlowService Setting Flow Controller's Node ID: <host-2>:8008
2018-10-25 07:12:53,340 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator This node is now connected to the cluster. Will no longer require election of DataFlow.
2018-10-25 07:12:53,341 INFO [main] o.apache.nifi.controller.FlowController Cluster State changed from Not Clustered to Clustered
What is especially weird is that there is no mention of host-1 whatsoever in this log, even though ZooKeeper is only running there.
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Wednesday, October 24, 2018 3:48 PM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
The part of the logs that might be suspicious is where it says "Cluster Coordinator is located at 0.0.0.0:11443".
I'm not totally sure, but I don't think it should have 0.0.0.0 for the address there.
What do you have set for nifi.cluster.node.address on each node? If it is not set can try setting it to the appropriate hostname for each cluster?
I'd be interested to see your nifi.properties for each node if you are able/willing to share that somehow.
On Wed, Oct 24, 2018 at 3:09 PM Viking K <cy...@hotmail.com>> wrote:
>
> Hi
> Don't know if this helps, but I run a 2 node cluster with 2 zookeepers.
> Did you do the zookeper myid assignment?
> Also have you set the flow election?
>
> I do it like this for my 2 instances,
>
> mkdir -p /nifi/state/zookeper/instance-1;echo 1
> >/nifi/state/zookeper/instance-1/myid
> mkdir -p /nifi/state/zookeper/instance-2;echo 2
> >/nifi/state/zookeper/instance-2/myid
>
> Then the cluster conf looks like this, its written in YML since I got a wrapper but its pretty self explaining.
>
> test:
> instances:
> host1:
> nifi.properties:
> nifi.cluster.node.address: 'host1.test.com'
> zookeeper.properties:
> clientPort: '2181'
> dataDir: '/nifi/state/zookeper/instance-1'
> host2:
> nifi.properties:
> nifi.cluster.node.address: 'host2.test.com'
>
> zookeeper.properties:
> clientPort: '2182'
> dataDir: '/nifi/state/zookeper/instance-2'
>
> common_configuration:
> nifi.properties:
> nifi.cluster.is.node: 'true'
> nifi.cluster.flow.election.max.candidates: '2'
> nifi.zookeeper.connect.string: 'host1.test.com:2181,host2.test.com:2182'
> nifi.state.management.embedded.zookeeper.start: 'true'
>
> bootstrap.conf:
> java.arg.2: '-Xms10g'
> java.arg.3: '-Xmx10g'
> run.as: 'nifi'
> zookeeper.properties:
> server.1: 'host1.test.com:2888:3888'
> server.2: 'host2.test.com:2889:3889'
>
> /Viking
> ________________________________
> From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>>
> Sent: Wednesday, October 24, 2018 5:47 PM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: RE: NiFi fails on cluster nodes
>
>
> I did what you suggested. There aren’t any errors in the log, although here is a warning:
>
>
>
> 2018-10-24 13:34:12,042 WARN [main]
> o.a.nifi.controller.StandardFlowService Failed to connect to cluster
> due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed
> unmarshalling 'CONNECTION_RESPONSE' protocol message from
> <host-2>/<host-2 IP address>:11443 due to:
> java.net.SocketTimeoutException: Read timed out
>
> 2018-10-24 13:34:12,049 INFO [main]
> o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster
> Coordinator is located at 0.0.0.0:11443; will use this address for
> sending heartbeat messages
>
> 2018-10-24 13:34:12,174 INFO [Process Cluster Protocol Request-2]
> o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from
> <host-2>:8008; responding with DataFlow that was elected
>
> 2018-10-24 13:34:12,175 INFO [Process Cluster Protocol Request-2]
> o.a.n.c.c.node.NodeClusterCoordinator Status of <host:8008 changed
> from NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING,
> updateId=61] to NodeConnectionStatus[nodeId=<host-2>:8008,
> state=CONNECTING, updateId=63]
>
>
>
> Please let me know if you want to see other cluster related INFO type log messages.
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Wednesday, October 24, 2018 12:08 PM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Is there anything interesting (errors/warnings) in nifi-app.log on host 2 during start up?
>
>
>
> Also, I'm not sure if this will do anything different, but you could try clearing the ZK state dir to make sure all the info in ZK is starting fresh...
>
>
>
> - Shutdown both nodes
>
> - Remove the directory nifi/state/zookeeper/version-2 on host 1 (not
> the whole ZK dir, just version-2)
>
> - Start nifi 1 and wait for it be up and running
>
> - Start nifi 2
>
>
>
> On Wed, Oct 24, 2018 at 11:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > Yes, that setting is the same for on both hosts. I attach the UI screenshots taken on those. Please note that host’s FQDNs have been removed.
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>>
>
> > Sent: Wednesday, October 24, 2018 9:25 AM
>
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > Many services can share a single ZooKeeper by segmenting their data under a specific root node.
>
> >
>
> >
>
> >
>
> > The root node is specified by nifi.zookeeper.root.node=/nifi so if those were different on each node then it would form separate clusters.
>
> >
>
> >
>
> >
>
> > Can you show screenshots of the cluster information from each node?
>
> >
>
> >
>
> >
>
> > May need to upload them somewhere and provide links here since attachments don't always make it through.
>
> >
>
> > On Wed, Oct 24, 2018 at 8:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > The ZooKeeper related settings in the nifi.properties files on both hosts are identical, with the exception of nifi.state.management.embedded.zookeeper.start, which is ‘true’ on host-1 and ‘false’ on host-2. Moreover, if I shut down NiFi on host-1, it crashes on host-2. Here is the message in the browser window:
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Action cannot be performed because there is currently no Cluster Coordinator elected. The request should be tried again after a moment, after a Cluster Coordinator has been automatically elected.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > I even went as far as commenting out the server.1 line in the zookeeper.properties file on host-1 before restarting both NiFi instances, which didn’t change the outcome.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > When I look at the NiFi Cluster information in the UI on host-1, it shows the status of the node “CONNECTED, PRIMARY, COORDINATOR”, whereas on host-2 just “CONNECTED”. I don’t know if this tells you anything.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > BTW, what does “a different location in the same ZK” mean?
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > Sent: Tuesday, October 23, 2018 3:02 PM
>
> >
>
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > The only way I could see that happening is if the ZK config on the second node pointed at a different ZK, or at a different location in the same ZK.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > For example, if node 1 had:
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.connect.string=node-1:2181
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.connect.timeout=3 secs
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.session.timeout=3 secs
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.root.node=/nifi
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Then node 2 should have exactly the same thing.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > If node 2 specified a different connect string, or a different root node, then it wouldn't know about the other node.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > That's exactly the case.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > >
>
> >
>
> > > > Sent: Tuesday, October 23, 2018 2:44 PM
>
> >
>
> > >
>
> >
>
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > >
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > It doesn't really make sense how the second node would form its own cluster.
>
> >
>
> > >
>
> >
>
> > > > On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > 2018-10-23 13:44:43,628 INFO
>
> >
>
> > >
>
> >
>
> > > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
>
> >
>
> > >
>
> >
>
> > > > > o.a.zookeeper.server.ZooKeeperServer Client attempting to
>
> >
>
> > > > > establish
>
> >
>
> > >
>
> >
>
> > > > > new session at /<host-2 IP address>:50412
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
>
> >
>
> > >
>
> >
>
> > > > > o.a.zookeeper.server.ZooKeeperServer Established session
>
> >
>
> > >
>
> >
>
> > > > > 0x166a1d139590002 with negotiated timeout 4000 for client
>
> > > > > /<host-2
>
> >
>
> > >
>
> >
>
> > > > > IP
>
> >
>
> > >
>
> >
>
> > > > > address>:50412
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > I apologize for bugging you with all this, converting our
>
> >
>
> > > > > standalone
>
> >
>
> > >
>
> >
>
> > > > > NiFi instances into cluster nodes turned out to be much more
>
> >
>
> > >
>
> >
>
> > > > > challenging than we had anticipated…
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > >
>
> >
>
> > > > > Sent: Tuesday, October 23, 2018 1:17 PM
>
> >
>
> > >
>
> >
>
> > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > >
>
> >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Sent: Tuesday, October 23, 2018 12:36 PM
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > That means the user representing host-1 does not have permissions to proxy.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > You can look in authorizations.xml on nifi-1 for a policy like:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > resource="/proxy" action="W">
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <user
>
> >
>
> > >
>
> >
>
> > > > > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > </policy>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > That user identifier should point to a user in users.xml like:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Government, C=US"/>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Hi Bryan,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · restarted one instance in the standalone mode
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · granted them the “proxy user requests” privileges
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · restarted the node on host-1
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Insufficient Permissions
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > The nifi-user.log also contains
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH,
>
> >
>
> > >
>
> >
>
> > > > > > > OU=HHS,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > O=U.S. Government, C=US
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > From your experience, what the most likely causes for this exception?
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Thank you,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Alexander
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Sent: Monday, October 22, 2018 1:25 PM
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Yes, to further clarify what I meant...
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
The part of the logs that might be suspicious is where it says
"Cluster Coordinator is located at 0.0.0.0:11443".
I'm not totally sure, but I don't think it should have 0.0.0.0 for the
address there.
What do you have set for nifi.cluster.node.address on each node? If it
is not set can try setting it to the appropriate hostname for each
cluster?
I'd be interested to see your nifi.properties for each node if you are
able/willing to share that somehow.
On Wed, Oct 24, 2018 at 3:09 PM Viking K <cy...@hotmail.com> wrote:
>
> Hi
> Don't know if this helps, but I run a 2 node cluster with 2 zookeepers.
> Did you do the zookeper myid assignment?
> Also have you set the flow election?
>
> I do it like this for my 2 instances,
>
> mkdir -p /nifi/state/zookeper/instance-1;echo 1 >/nifi/state/zookeper/instance-1/myid
> mkdir -p /nifi/state/zookeper/instance-2;echo 2 >/nifi/state/zookeper/instance-2/myid
>
> Then the cluster conf looks like this, its written in YML since I got a wrapper but its pretty self explaining.
>
> test:
> instances:
> host1:
> nifi.properties:
> nifi.cluster.node.address: 'host1.test.com'
> zookeeper.properties:
> clientPort: '2181'
> dataDir: '/nifi/state/zookeper/instance-1'
> host2:
> nifi.properties:
> nifi.cluster.node.address: 'host2.test.com'
>
> zookeeper.properties:
> clientPort: '2182'
> dataDir: '/nifi/state/zookeper/instance-2'
>
> common_configuration:
> nifi.properties:
> nifi.cluster.is.node: 'true'
> nifi.cluster.flow.election.max.candidates: '2'
> nifi.zookeeper.connect.string: 'host1.test.com:2181,host2.test.com:2182'
> nifi.state.management.embedded.zookeeper.start: 'true'
>
> bootstrap.conf:
> java.arg.2: '-Xms10g'
> java.arg.3: '-Xmx10g'
> run.as: 'nifi'
> zookeeper.properties:
> server.1: 'host1.test.com:2888:3888'
> server.2: 'host2.test.com:2889:3889'
>
> /Viking
> ________________________________
> From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
> Sent: Wednesday, October 24, 2018 5:47 PM
> To: users@nifi.apache.org
> Subject: RE: NiFi fails on cluster nodes
>
>
> I did what you suggested. There aren’t any errors in the log, although here is a warning:
>
>
>
> 2018-10-24 13:34:12,042 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from <host-2>/<host-2 IP address>:11443 due to: java.net.SocketTimeoutException: Read timed out
>
> 2018-10-24 13:34:12,049 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 0.0.0.0:11443; will use this address for sending heartbeat messages
>
> 2018-10-24 13:34:12,174 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from <host-2>:8008; responding with DataFlow that was elected
>
> 2018-10-24 13:34:12,175 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Status of <host:8008 changed from NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=61] to NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=63]
>
>
>
> Please let me know if you want to see other cluster related INFO type log messages.
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Wednesday, October 24, 2018 12:08 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Is there anything interesting (errors/warnings) in nifi-app.log on host 2 during start up?
>
>
>
> Also, I'm not sure if this will do anything different, but you could try clearing the ZK state dir to make sure all the info in ZK is starting fresh...
>
>
>
> - Shutdown both nodes
>
> - Remove the directory nifi/state/zookeeper/version-2 on host 1 (not the whole ZK dir, just version-2)
>
> - Start nifi 1 and wait for it be up and running
>
> - Start nifi 2
>
>
>
> On Wed, Oct 24, 2018 at 11:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > Yes, that setting is the same for on both hosts. I attach the UI screenshots taken on those. Please note that host’s FQDNs have been removed.
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Wednesday, October 24, 2018 9:25 AM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > Many services can share a single ZooKeeper by segmenting their data under a specific root node.
>
> >
>
> >
>
> >
>
> > The root node is specified by nifi.zookeeper.root.node=/nifi so if those were different on each node then it would form separate clusters.
>
> >
>
> >
>
> >
>
> > Can you show screenshots of the cluster information from each node?
>
> >
>
> >
>
> >
>
> > May need to upload them somewhere and provide links here since attachments don't always make it through.
>
> >
>
> > On Wed, Oct 24, 2018 at 8:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > The ZooKeeper related settings in the nifi.properties files on both hosts are identical, with the exception of nifi.state.management.embedded.zookeeper.start, which is ‘true’ on host-1 and ‘false’ on host-2. Moreover, if I shut down NiFi on host-1, it crashes on host-2. Here is the message in the browser window:
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Action cannot be performed because there is currently no Cluster Coordinator elected. The request should be tried again after a moment, after a Cluster Coordinator has been automatically elected.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > I even went as far as commenting out the server.1 line in the zookeeper.properties file on host-1 before restarting both NiFi instances, which didn’t change the outcome.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > When I look at the NiFi Cluster information in the UI on host-1, it shows the status of the node “CONNECTED, PRIMARY, COORDINATOR”, whereas on host-2 just “CONNECTED”. I don’t know if this tells you anything.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > BTW, what does “a different location in the same ZK” mean?
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > Sent: Tuesday, October 23, 2018 3:02 PM
>
> >
>
> > > To: users@nifi.apache.org
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > The only way I could see that happening is if the ZK config on the second node pointed at a different ZK, or at a different location in the same ZK.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > For example, if node 1 had:
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.connect.string=node-1:2181
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.connect.timeout=3 secs
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.session.timeout=3 secs
>
> >
>
> > >
>
> >
>
> > > nifi.zookeeper.root.node=/nifi
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Then node 2 should have exactly the same thing.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > If node 2 specified a different connect string, or a different root node, then it wouldn't know about the other node.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > That's exactly the case.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > >
>
> >
>
> > > > Sent: Tuesday, October 23, 2018 2:44 PM
>
> >
>
> > >
>
> >
>
> > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > It doesn't really make sense how the second node would form its own cluster.
>
> >
>
> > >
>
> >
>
> > > > On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > 2018-10-23 13:44:43,628 INFO
>
> >
>
> > >
>
> >
>
> > > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
>
> >
>
> > >
>
> >
>
> > > > > o.a.zookeeper.server.ZooKeeperServer Client attempting to
>
> >
>
> > > > > establish
>
> >
>
> > >
>
> >
>
> > > > > new session at /<host-2 IP address>:50412
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
>
> >
>
> > >
>
> >
>
> > > > > o.a.zookeeper.server.ZooKeeperServer Established session
>
> >
>
> > >
>
> >
>
> > > > > 0x166a1d139590002 with negotiated timeout 4000 for client
>
> > > > > /<host-2
>
> >
>
> > >
>
> >
>
> > > > > IP
>
> >
>
> > >
>
> >
>
> > > > > address>:50412
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > I apologize for bugging you with all this, converting our
>
> >
>
> > > > > standalone
>
> >
>
> > >
>
> >
>
> > > > > NiFi instances into cluster nodes turned out to be much more
>
> >
>
> > >
>
> >
>
> > > > > challenging than we had anticipated…
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > >
>
> >
>
> > > > > Sent: Tuesday, October 23, 2018 1:17 PM
>
> >
>
> > >
>
> >
>
> > > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Sent: Tuesday, October 23, 2018 12:36 PM
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > That means the user representing host-1 does not have permissions to proxy.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > You can look in authorizations.xml on nifi-1 for a policy like:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > resource="/proxy" action="W">
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <user
>
> >
>
> > >
>
> >
>
> > > > > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > </policy>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > That user identifier should point to a user in users.xml like:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Government, C=US"/>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Hi Bryan,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · restarted one instance in the standalone mode
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · granted them the “proxy user requests” privileges
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > · restarted the node on host-1
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Insufficient Permissions
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > The nifi-user.log also contains
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH,
>
> >
>
> > >
>
> >
>
> > > > > > > OU=HHS,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > O=U.S. Government, C=US
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > From your experience, what the most likely causes for this exception?
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Thank you,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Alexander
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Sent: Monday, October 22, 2018 1:25 PM
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > > Yes, to further clarify what I meant...
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > >
>
> >
Re: NiFi fails on cluster nodes
Posted by Viking K <cy...@hotmail.com>.
Hi
Don't know if this helps, but I run a 2 node cluster with 2 zookeepers.
Did you do the zookeper myid assignment?
Also have you set the flow election?
I do it like this for my 2 instances,
mkdir -p /nifi/state/zookeper/instance-1;echo 1 >/nifi/state/zookeper/instance-1/myid
mkdir -p /nifi/state/zookeper/instance-2;echo 2 >/nifi/state/zookeper/instance-2/myid
Then the cluster conf looks like this, its written in YML since I got a wrapper but its pretty self explaining.
test:
instances:
host1:
nifi.properties:
nifi.cluster.node.address: 'host1.test.com'
zookeeper.properties:
clientPort: '2181'
dataDir: '/nifi/state/zookeper/instance-1'
host2:
nifi.properties:
nifi.cluster.node.address: 'host2.test.com'
zookeeper.properties:
clientPort: '2182'
dataDir: '/nifi/state/zookeper/instance-2'
common_configuration:
nifi.properties:
nifi.cluster.is.node: 'true'
nifi.cluster.flow.election.max.candidates: '2'
nifi.zookeeper.connect.string: 'host1.test.com:2181,host2.test.com:2182'
nifi.state.management.embedded.zookeeper.start: 'true'
bootstrap.conf:
java.arg.2: '-Xms10g'
java.arg.3: '-Xmx10g'
run.as: 'nifi'
zookeeper.properties:
server.1: 'host1.test.com:2888:3888'
server.2: 'host2.test.com:2889:3889'
/Viking
________________________________
From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
Sent: Wednesday, October 24, 2018 5:47 PM
To: users@nifi.apache.org
Subject: RE: NiFi fails on cluster nodes
I did what you suggested. There aren’t any errors in the log, although here is a warning:
2018-10-24 13:34:12,042 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from <host-2>/<host-2 IP address>:11443 due to: java.net.SocketTimeoutException: Read timed out
2018-10-24 13:34:12,049 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 0.0.0.0:11443; will use this address for sending heartbeat messages
2018-10-24 13:34:12,174 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from <host-2>:8008; responding with DataFlow that was elected
2018-10-24 13:34:12,175 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Status of <host:8008 changed from NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=61] to NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=63]
Please let me know if you want to see other cluster related INFO type log messages.
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Wednesday, October 24, 2018 12:08 PM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
Is there anything interesting (errors/warnings) in nifi-app.log on host 2 during start up?
Also, I'm not sure if this will do anything different, but you could try clearing the ZK state dir to make sure all the info in ZK is starting fresh...
- Shutdown both nodes
- Remove the directory nifi/state/zookeeper/version-2 on host 1 (not the whole ZK dir, just version-2)
- Start nifi 1 and wait for it be up and running
- Start nifi 2
On Wed, Oct 24, 2018 at 11:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> Yes, that setting is the same for on both hosts. I attach the UI screenshots taken on those. Please note that host’s FQDNs have been removed.
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Wednesday, October 24, 2018 9:25 AM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Many services can share a single ZooKeeper by segmenting their data under a specific root node.
>
>
>
> The root node is specified by nifi.zookeeper.root.node=/nifi so if those were different on each node then it would form separate clusters.
>
>
>
> Can you show screenshots of the cluster information from each node?
>
>
>
> May need to upload them somewhere and provide links here since attachments don't always make it through.
>
> On Wed, Oct 24, 2018 at 8:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > The ZooKeeper related settings in the nifi.properties files on both hosts are identical, with the exception of nifi.state.management.embedded.zookeeper.start, which is ‘true’ on host-1 and ‘false’ on host-2. Moreover, if I shut down NiFi on host-1, it crashes on host-2. Here is the message in the browser window:
>
> >
>
> >
>
> >
>
> > Action cannot be performed because there is currently no Cluster Coordinator elected. The request should be tried again after a moment, after a Cluster Coordinator has been automatically elected.
>
> >
>
> >
>
> >
>
> > I even went as far as commenting out the server.1 line in the zookeeper.properties file on host-1 before restarting both NiFi instances, which didn’t change the outcome.
>
> >
>
> >
>
> >
>
> > When I look at the NiFi Cluster information in the UI on host-1, it shows the status of the node “CONNECTED, PRIMARY, COORDINATOR”, whereas on host-2 just “CONNECTED”. I don’t know if this tells you anything.
>
> >
>
> >
>
> >
>
> > BTW, what does “a different location in the same ZK” mean?
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>>
>
> > Sent: Tuesday, October 23, 2018 3:02 PM
>
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > The only way I could see that happening is if the ZK config on the second node pointed at a different ZK, or at a different location in the same ZK.
>
> >
>
> >
>
> >
>
> > For example, if node 1 had:
>
> >
>
> >
>
> >
>
> > nifi.zookeeper.connect.string=node-1:2181
>
> >
>
> > nifi.zookeeper.connect.timeout=3 secs
>
> >
>
> > nifi.zookeeper.session.timeout=3 secs
>
> >
>
> > nifi.zookeeper.root.node=/nifi
>
> >
>
> >
>
> >
>
> > Then node 2 should have exactly the same thing.
>
> >
>
> >
>
> >
>
> > If node 2 specified a different connect string, or a different root node, then it wouldn't know about the other node.
>
> >
>
> >
>
> >
>
> > On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > That's exactly the case.
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > Sent: Tuesday, October 23, 2018 2:44 PM
>
> >
>
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> >
>
> > >
>
> >
>
> > > It doesn't really make sense how the second node would form its own cluster.
>
> >
>
> > > On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > 2018-10-23 13:44:43,628 INFO
>
> >
>
> > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
>
> >
>
> > > > o.a.zookeeper.server.ZooKeeperServer Client attempting to
>
> > > > establish
>
> >
>
> > > > new session at /<host-2 IP address>:50412
>
> >
>
> > > >
>
> >
>
> > > > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
>
> >
>
> > > > o.a.zookeeper.server.ZooKeeperServer Established session
>
> >
>
> > > > 0x166a1d139590002 with negotiated timeout 4000 for client
> > > > /<host-2
>
> >
>
> > > > IP
>
> >
>
> > > > address>:50412
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > I apologize for bugging you with all this, converting our
>
> > > > standalone
>
> >
>
> > > > NiFi instances into cluster nodes turned out to be much more
>
> >
>
> > > > challenging than we had anticipated…
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > > Sent: Tuesday, October 23, 2018 1:17 PM
>
> >
>
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> >
>
> > > >
>
> >
>
> > > > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > -----Original Message-----
>
> >
>
> > > >
>
> >
>
> > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > >
>
> >
>
> > > > > Sent: Tuesday, October 23, 2018 12:36 PM
>
> >
>
> > > >
>
> >
>
> > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > >
>
> >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > That means the user representing host-1 does not have permissions to proxy.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > You can look in authorizations.xml on nifi-1 for a policy like:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > resource="/proxy" action="W">
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <user
>
> >
>
> > > > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > </policy>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > That user identifier should point to a user in users.xml like:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > Government, C=US"/>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Hi Bryan,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · restarted one instance in the standalone mode
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · granted them the “proxy user requests” privileges
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · restarted the node on host-1
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Insufficient Permissions
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > The nifi-user.log also contains
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH,
>
> >
>
> > > > > > OU=HHS,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > O=U.S. Government, C=US
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > From your experience, what the most likely causes for this exception?
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Thank you,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Alexander
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > -----Original Message-----
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Sent: Monday, October 22, 2018 1:25 PM
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Yes, to further clarify what I meant...
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > - In standalone mode, use the UI to add users for the DN's
> > > > > > of
>
> >
>
> > > > > > the
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > server certificates (CN=nifi-node-1, OU=NIFI,
> > > > > > CN=nifi-node-2,
>
> >
>
> > > >
>
> >
>
> > > > > > OU=NIFI)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > - In the UI, grant those users Write access to "Proxy"
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > - Convert to a cluster and keep your same authorizers.xml,
>
> >
>
> > > >
>
> >
>
> > > > > > users.xml,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > and authorizations.xml when you setup your cluster, this way
>
> > > > > > all
>
> >
>
> > > >
>
> >
>
> > > > > > your
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > users and policies are already setup and the Initial Admin
> > > > > > and
>
> >
>
> > > > > > Node
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Identities are not needed
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > -----Original Message-----
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Sent: Monday, October 22, 2018 12:48 PM
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > This blog post shows how to setup a secure 2 node cluster:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > https://bryanbende.com/development/2016/08/17/apache-nifi-
> > > > > > > 1-
>
> > > > > > > 0-
>
> >
>
> > > > > > > 0-
>
> >
>
> > > > > > > au
>
> >
>
> > > >
>
> >
>
> > > > > > > th
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > or
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > ization-and-multi-tenancy
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <authorizer>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <identifier>file-provider</identifier>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > > > > > <class>org.apache.nifi.authorization.FileAuthorizer</class
> > > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Authorizations
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > File">./conf/authorizations.xml</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Users
> > > > > > > File">./conf/users.xml</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Initial Admin Identity">CN=bbende,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > OU=ApacheNiFi</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Legacy Authorized Users
>
> > > > > > > File"></property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Node Identity 1">CN=localhost,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > OU=NIFI</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > </authorizer>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
I did what you suggested. There aren’t any errors in the log, although here is a warning:
2018-10-24 13:34:12,042 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed unmarshalling 'CONNECTION_RESPONSE' protocol message from <host-2>/<host-2 IP address>:11443 due to: java.net.SocketTimeoutException: Read timed out
2018-10-24 13:34:12,049 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at 0.0.0.0:11443; will use this address for sending heartbeat messages
2018-10-24 13:34:12,174 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Received Connection Request from <host-2>:8008; responding with DataFlow that was elected
2018-10-24 13:34:12,175 INFO [Process Cluster Protocol Request-2] o.a.n.c.c.node.NodeClusterCoordinator Status of <host:8008 changed from NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=61] to NodeConnectionStatus[nodeId=<host-2>:8008, state=CONNECTING, updateId=63]
Please let me know if you want to see other cluster related INFO type log messages.
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Wednesday, October 24, 2018 12:08 PM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
Is there anything interesting (errors/warnings) in nifi-app.log on host 2 during start up?
Also, I'm not sure if this will do anything different, but you could try clearing the ZK state dir to make sure all the info in ZK is starting fresh...
- Shutdown both nodes
- Remove the directory nifi/state/zookeeper/version-2 on host 1 (not the whole ZK dir, just version-2)
- Start nifi 1 and wait for it be up and running
- Start nifi 2
On Wed, Oct 24, 2018 at 11:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> Yes, that setting is the same for on both hosts. I attach the UI screenshots taken on those. Please note that host’s FQDNs have been removed.
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Wednesday, October 24, 2018 9:25 AM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Many services can share a single ZooKeeper by segmenting their data under a specific root node.
>
>
>
> The root node is specified by nifi.zookeeper.root.node=/nifi so if those were different on each node then it would form separate clusters.
>
>
>
> Can you show screenshots of the cluster information from each node?
>
>
>
> May need to upload them somewhere and provide links here since attachments don't always make it through.
>
> On Wed, Oct 24, 2018 at 8:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > The ZooKeeper related settings in the nifi.properties files on both hosts are identical, with the exception of nifi.state.management.embedded.zookeeper.start, which is ‘true’ on host-1 and ‘false’ on host-2. Moreover, if I shut down NiFi on host-1, it crashes on host-2. Here is the message in the browser window:
>
> >
>
> >
>
> >
>
> > Action cannot be performed because there is currently no Cluster Coordinator elected. The request should be tried again after a moment, after a Cluster Coordinator has been automatically elected.
>
> >
>
> >
>
> >
>
> > I even went as far as commenting out the server.1 line in the zookeeper.properties file on host-1 before restarting both NiFi instances, which didn’t change the outcome.
>
> >
>
> >
>
> >
>
> > When I look at the NiFi Cluster information in the UI on host-1, it shows the status of the node “CONNECTED, PRIMARY, COORDINATOR”, whereas on host-2 just “CONNECTED”. I don’t know if this tells you anything.
>
> >
>
> >
>
> >
>
> > BTW, what does “a different location in the same ZK” mean?
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>>
>
> > Sent: Tuesday, October 23, 2018 3:02 PM
>
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > The only way I could see that happening is if the ZK config on the second node pointed at a different ZK, or at a different location in the same ZK.
>
> >
>
> >
>
> >
>
> > For example, if node 1 had:
>
> >
>
> >
>
> >
>
> > nifi.zookeeper.connect.string=node-1:2181
>
> >
>
> > nifi.zookeeper.connect.timeout=3 secs
>
> >
>
> > nifi.zookeeper.session.timeout=3 secs
>
> >
>
> > nifi.zookeeper.root.node=/nifi
>
> >
>
> >
>
> >
>
> > Then node 2 should have exactly the same thing.
>
> >
>
> >
>
> >
>
> > If node 2 specified a different connect string, or a different root node, then it wouldn't know about the other node.
>
> >
>
> >
>
> >
>
> > On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > That's exactly the case.
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > Sent: Tuesday, October 23, 2018 2:44 PM
>
> >
>
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> >
>
> > >
>
> >
>
> > > It doesn't really make sense how the second node would form its own cluster.
>
> >
>
> > > On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > 2018-10-23 13:44:43,628 INFO
>
> >
>
> > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
>
> >
>
> > > > o.a.zookeeper.server.ZooKeeperServer Client attempting to
>
> > > > establish
>
> >
>
> > > > new session at /<host-2 IP address>:50412
>
> >
>
> > > >
>
> >
>
> > > > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
>
> >
>
> > > > o.a.zookeeper.server.ZooKeeperServer Established session
>
> >
>
> > > > 0x166a1d139590002 with negotiated timeout 4000 for client
> > > > /<host-2
>
> >
>
> > > > IP
>
> >
>
> > > > address>:50412
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > I apologize for bugging you with all this, converting our
>
> > > > standalone
>
> >
>
> > > > NiFi instances into cluster nodes turned out to be much more
>
> >
>
> > > > challenging than we had anticipated…
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > > Sent: Tuesday, October 23, 2018 1:17 PM
>
> >
>
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> >
>
> > > >
>
> >
>
> > > > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > -----Original Message-----
>
> >
>
> > > >
>
> >
>
> > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > >
>
> >
>
> > > > > Sent: Tuesday, October 23, 2018 12:36 PM
>
> >
>
> > > >
>
> >
>
> > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > >
>
> >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > That means the user representing host-1 does not have permissions to proxy.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > You can look in authorizations.xml on nifi-1 for a policy like:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > resource="/proxy" action="W">
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <user
>
> >
>
> > > > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > </policy>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > That user identifier should point to a user in users.xml like:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > Government, C=US"/>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Hi Bryan,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · restarted one instance in the standalone mode
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · granted them the “proxy user requests” privileges
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · restarted the node on host-1
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Insufficient Permissions
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > The nifi-user.log also contains
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH,
>
> >
>
> > > > > > OU=HHS,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > O=U.S. Government, C=US
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > From your experience, what the most likely causes for this exception?
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Thank you,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Alexander
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > -----Original Message-----
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Sent: Monday, October 22, 2018 1:25 PM
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Yes, to further clarify what I meant...
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > - In standalone mode, use the UI to add users for the DN's
> > > > > > of
>
> >
>
> > > > > > the
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > server certificates (CN=nifi-node-1, OU=NIFI,
> > > > > > CN=nifi-node-2,
>
> >
>
> > > >
>
> >
>
> > > > > > OU=NIFI)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > - In the UI, grant those users Write access to "Proxy"
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > - Convert to a cluster and keep your same authorizers.xml,
>
> >
>
> > > >
>
> >
>
> > > > > > users.xml,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > and authorizations.xml when you setup your cluster, this way
>
> > > > > > all
>
> >
>
> > > >
>
> >
>
> > > > > > your
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > users and policies are already setup and the Initial Admin
> > > > > > and
>
> >
>
> > > > > > Node
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Identities are not needed
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > -----Original Message-----
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Sent: Monday, October 22, 2018 12:48 PM
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > This blog post shows how to setup a secure 2 node cluster:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > https://bryanbende.com/development/2016/08/17/apache-nifi-
> > > > > > > 1-
>
> > > > > > > 0-
>
> >
>
> > > > > > > 0-
>
> >
>
> > > > > > > au
>
> >
>
> > > >
>
> >
>
> > > > > > > th
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > or
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > ization-and-multi-tenancy
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <authorizer>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <identifier>file-provider</identifier>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > > > > > <class>org.apache.nifi.authorization.FileAuthorizer</class
> > > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Authorizations
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > File">./conf/authorizations.xml</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Users
> > > > > > > File">./conf/users.xml</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Initial Admin Identity">CN=bbende,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > OU=ApacheNiFi</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Legacy Authorized Users
>
> > > > > > > File"></property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Node Identity 1">CN=localhost,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > OU=NIFI</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > </authorizer>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
Is there anything interesting (errors/warnings) in nifi-app.log on
host 2 during start up?
Also, I'm not sure if this will do anything different, but you could
try clearing the ZK state dir to make sure all the info in ZK is
starting fresh...
- Shutdown both nodes
- Remove the directory nifi/state/zookeeper/version-2 on host 1 (not
the whole ZK dir, just version-2)
- Start nifi 1 and wait for it be up and running
- Start nifi 2
On Wed, Oct 24, 2018 at 11:18 AM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> Yes, that setting is the same for on both hosts. I attach the UI screenshots taken on those. Please note that host’s FQDNs have been removed.
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Wednesday, October 24, 2018 9:25 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Many services can share a single ZooKeeper by segmenting their data under a specific root node.
>
>
>
> The root node is specified by nifi.zookeeper.root.node=/nifi so if those were different on each node then it would form separate clusters.
>
>
>
> Can you show screenshots of the cluster information from each node?
>
>
>
> May need to upload them somewhere and provide links here since attachments don't always make it through.
>
> On Wed, Oct 24, 2018 at 8:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > The ZooKeeper related settings in the nifi.properties files on both hosts are identical, with the exception of nifi.state.management.embedded.zookeeper.start, which is ‘true’ on host-1 and ‘false’ on host-2. Moreover, if I shut down NiFi on host-1, it crashes on host-2. Here is the message in the browser window:
>
> >
>
> >
>
> >
>
> > Action cannot be performed because there is currently no Cluster Coordinator elected. The request should be tried again after a moment, after a Cluster Coordinator has been automatically elected.
>
> >
>
> >
>
> >
>
> > I even went as far as commenting out the server.1 line in the zookeeper.properties file on host-1 before restarting both NiFi instances, which didn’t change the outcome.
>
> >
>
> >
>
> >
>
> > When I look at the NiFi Cluster information in the UI on host-1, it shows the status of the node “CONNECTED, PRIMARY, COORDINATOR”, whereas on host-2 just “CONNECTED”. I don’t know if this tells you anything.
>
> >
>
> >
>
> >
>
> > BTW, what does “a different location in the same ZK” mean?
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Tuesday, October 23, 2018 3:02 PM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > The only way I could see that happening is if the ZK config on the second node pointed at a different ZK, or at a different location in the same ZK.
>
> >
>
> >
>
> >
>
> > For example, if node 1 had:
>
> >
>
> >
>
> >
>
> > nifi.zookeeper.connect.string=node-1:2181
>
> >
>
> > nifi.zookeeper.connect.timeout=3 secs
>
> >
>
> > nifi.zookeeper.session.timeout=3 secs
>
> >
>
> > nifi.zookeeper.root.node=/nifi
>
> >
>
> >
>
> >
>
> > Then node 2 should have exactly the same thing.
>
> >
>
> >
>
> >
>
> > If node 2 specified a different connect string, or a different root node, then it wouldn't know about the other node.
>
> >
>
> >
>
> >
>
> > On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > That's exactly the case.
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > Sent: Tuesday, October 23, 2018 2:44 PM
>
> >
>
> > > To: users@nifi.apache.org
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> >
>
> > >
>
> >
>
> > > It doesn't really make sense how the second node would form its own cluster.
>
> >
>
> > > On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >
>
> >
>
> > > > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > 2018-10-23 13:44:43,628 INFO
>
> >
>
> > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
>
> >
>
> > > > o.a.zookeeper.server.ZooKeeperServer Client attempting to
>
> > > > establish
>
> >
>
> > > > new session at /<host-2 IP address>:50412
>
> >
>
> > > >
>
> >
>
> > > > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
>
> >
>
> > > > o.a.zookeeper.server.ZooKeeperServer Established session
>
> >
>
> > > > 0x166a1d139590002 with negotiated timeout 4000 for client /<host-2
>
> >
>
> > > > IP
>
> >
>
> > > > address>:50412
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > I apologize for bugging you with all this, converting our
>
> > > > standalone
>
> >
>
> > > > NiFi instances into cluster nodes turned out to be much more
>
> >
>
> > > > challenging than we had anticipated…
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > > Sent: Tuesday, October 23, 2018 1:17 PM
>
> >
>
> > > > To: users@nifi.apache.org
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> >
>
> > > >
>
> >
>
> > > > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > -----Original Message-----
>
> >
>
> > > >
>
> >
>
> > > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > >
>
> >
>
> > > > > Sent: Tuesday, October 23, 2018 12:36 PM
>
> >
>
> > > >
>
> >
>
> > > > > To: users@nifi.apache.org
>
> >
>
> > > >
>
> >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > That means the user representing host-1 does not have permissions to proxy.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > You can look in authorizations.xml on nifi-1 for a policy like:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > resource="/proxy" action="W">
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <user
>
> >
>
> > > > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > </policy>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > That user identifier should point to a user in users.xml like:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > Government, C=US"/>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Hi Bryan,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · restarted one instance in the standalone mode
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · granted them the “proxy user requests” privileges
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > · restarted the node on host-1
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Insufficient Permissions
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > The nifi-user.log also contains
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH,
>
> >
>
> > > > > > OU=HHS,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > O=U.S. Government, C=US
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > From your experience, what the most likely causes for this exception?
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Thank you,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Alexander
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > -----Original Message-----
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Sent: Monday, October 22, 2018 1:25 PM
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > To: users@nifi.apache.org
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Yes, to further clarify what I meant...
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > - In standalone mode, use the UI to add users for the DN's of
>
> >
>
> > > > > > the
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2,
>
> >
>
> > > >
>
> >
>
> > > > > > OU=NIFI)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > - In the UI, grant those users Write access to "Proxy"
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > - Convert to a cluster and keep your same authorizers.xml,
>
> >
>
> > > >
>
> >
>
> > > > > > users.xml,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > and authorizations.xml when you setup your cluster, this way
>
> > > > > > all
>
> >
>
> > > >
>
> >
>
> > > > > > your
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > users and policies are already setup and the Initial Admin and
>
> >
>
> > > > > > Node
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > Identities are not needed
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > -----Original Message-----
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Sent: Monday, October 22, 2018 12:48 PM
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > To: users@nifi.apache.org
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > This blog post shows how to setup a secure 2 node cluster:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-
>
> > > > > > > 0-
>
> >
>
> > > > > > > 0-
>
> >
>
> > > > > > > au
>
> >
>
> > > >
>
> >
>
> > > > > > > th
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > or
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > ization-and-multi-tenancy
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <authorizer>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <identifier>file-provider</identifier>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > >
>
> >
>
> > > > > > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Authorizations
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > File">./conf/authorizations.xml</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Users File">./conf/users.xml</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Initial Admin Identity">CN=bbende,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > OU=ApacheNiFi</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Legacy Authorized Users
>
> > > > > > > File"></property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > <property name="Node Identity 1">CN=localhost,
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > OU=NIFI</property>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > > > </authorizer>
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Yes, that setting is the same for on both hosts. I attach the UI screenshots taken on those. Please note that host’s FQDNs have been removed.
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Wednesday, October 24, 2018 9:25 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
Many services can share a single ZooKeeper by segmenting their data under a specific root node.
The root node is specified by nifi.zookeeper.root.node=/nifi so if those were different on each node then it would form separate clusters.
Can you show screenshots of the cluster information from each node?
May need to upload them somewhere and provide links here since attachments don't always make it through.
On Wed, Oct 24, 2018 at 8:18 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> The ZooKeeper related settings in the nifi.properties files on both hosts are identical, with the exception of nifi.state.management.embedded.zookeeper.start, which is ‘true’ on host-1 and ‘false’ on host-2. Moreover, if I shut down NiFi on host-1, it crashes on host-2. Here is the message in the browser window:
>
>
>
> Action cannot be performed because there is currently no Cluster Coordinator elected. The request should be tried again after a moment, after a Cluster Coordinator has been automatically elected.
>
>
>
> I even went as far as commenting out the server.1 line in the zookeeper.properties file on host-1 before restarting both NiFi instances, which didn’t change the outcome.
>
>
>
> When I look at the NiFi Cluster information in the UI on host-1, it shows the status of the node “CONNECTED, PRIMARY, COORDINATOR”, whereas on host-2 just “CONNECTED”. I don’t know if this tells you anything.
>
>
>
> BTW, what does “a different location in the same ZK” mean?
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Tuesday, October 23, 2018 3:02 PM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> The only way I could see that happening is if the ZK config on the second node pointed at a different ZK, or at a different location in the same ZK.
>
>
>
> For example, if node 1 had:
>
>
>
> nifi.zookeeper.connect.string=node-1:2181
>
> nifi.zookeeper.connect.timeout=3 secs
>
> nifi.zookeeper.session.timeout=3 secs
>
> nifi.zookeeper.root.node=/nifi
>
>
>
> Then node 2 should have exactly the same thing.
>
>
>
> If node 2 specified a different connect string, or a different root node, then it wouldn't know about the other node.
>
>
>
> On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > That's exactly the case.
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>>
>
> > Sent: Tuesday, October 23, 2018 2:44 PM
>
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> >
>
> > It doesn't really make sense how the second node would form its own cluster.
>
> > On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
> > >
>
> > >
>
> > >
>
> > > 2018-10-23 13:44:43,628 INFO
>
> > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
>
> > > o.a.zookeeper.server.ZooKeeperServer Client attempting to
> > > establish
>
> > > new session at /<host-2 IP address>:50412
>
> > >
>
> > > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
>
> > > o.a.zookeeper.server.ZooKeeperServer Established session
>
> > > 0x166a1d139590002 with negotiated timeout 4000 for client /<host-2
>
> > > IP
>
> > > address>:50412
>
> > >
>
> > >
>
> > >
>
> > > I apologize for bugging you with all this, converting our
> > > standalone
>
> > > NiFi instances into cluster nodes turned out to be much more
>
> > > challenging than we had anticipated…
>
> > >
>
> > >
>
> > >
>
> > > -----Original Message-----
>
> > > From: Bryan Bende <bb...@gmail.com>>
>
> > > Sent: Tuesday, October 23, 2018 1:17 PM
>
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > >
>
> > >
>
> > > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> > >
>
> > > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > -----Original Message-----
>
> > >
>
> > > > From: Bryan Bende <bb...@gmail.com>>
>
> > >
>
> > > > Sent: Tuesday, October 23, 2018 12:36 PM
>
> > >
>
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > That means the user representing host-1 does not have permissions to proxy.
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > You can look in authorizations.xml on nifi-1 for a policy like:
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> > >
>
> > > >
>
> > >
>
> > > > resource="/proxy" action="W">
>
> > >
>
> > > >
>
> > >
>
> > > > <user
>
> > > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> > >
>
> > > >
>
> > >
>
> > > > </policy>
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > That user identifier should point to a user in users.xml like:
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> > >
>
> > > >
>
> > >
>
> > > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> > >
>
> > > >
>
> > >
>
> > > > Government, C=US"/>
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Hi Bryan,
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · restarted one instance in the standalone mode
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · granted them the “proxy user requests” privileges
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · restarted the node on host-1
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Insufficient Permissions
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > The nifi-user.log also contains
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> > >
>
> > > >
>
> > >
>
> > > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> > >
>
> > > >
>
> > >
>
> > > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH,
>
> > > > > OU=HHS,
>
> > >
>
> > > >
>
> > >
>
> > > > > O=U.S. Government, C=US
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > From your experience, what the most likely causes for this exception?
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Thank you,
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Alexander
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > -----Original Message-----
>
> > >
>
> > > >
>
> > >
>
> > > > > From: Bryan Bende <bb...@gmail.com>>
>
> > >
>
> > > >
>
> > >
>
> > > > > Sent: Monday, October 22, 2018 1:25 PM
>
> > >
>
> > > >
>
> > >
>
> > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >
>
> > > >
>
> > >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Yes, to further clarify what I meant...
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > - In standalone mode, use the UI to add users for the DN's of
>
> > > > > the
>
> > >
>
> > > >
>
> > >
>
> > > > > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2,
>
> > >
>
> > > > > OU=NIFI)
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > - In the UI, grant those users Write access to "Proxy"
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > - Convert to a cluster and keep your same authorizers.xml,
>
> > >
>
> > > > > users.xml,
>
> > >
>
> > > >
>
> > >
>
> > > > > and authorizations.xml when you setup your cluster, this way
> > > > > all
>
> > >
>
> > > > > your
>
> > >
>
> > > >
>
> > >
>
> > > > > users and policies are already setup and the Initial Admin and
>
> > > > > Node
>
> > >
>
> > > >
>
> > >
>
> > > > > Identities are not needed
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > -----Original Message-----
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Sent: Monday, October 22, 2018 12:48 PM
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > This blog post shows how to setup a secure 2 node cluster:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-
> > > > > > 0-
>
> > > > > > 0-
>
> > > > > > au
>
> > >
>
> > > > > > th
>
> > >
>
> > > >
>
> > >
>
> > > > > > or
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > ization-and-multi-tenancy
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <authorizer>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <identifier>file-provider</identifier>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > > > > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Authorizations
>
> > >
>
> > > >
>
> > >
>
> > > > > > File">./conf/authorizations.xml</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Users File">./conf/users.xml</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial Admin Identity">CN=bbende,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=ApacheNiFi</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Legacy Authorized Users
> > > > > > File"></property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Node Identity 1">CN=localhost,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > </authorizer>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <userGroupProvider>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <identifier>file-user-group-provider</identifier>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <class>org.apache.nifi.authorization.FileUserGroupProvider</
> > > > > > cl
>
> > > > > > as
>
> > > > > > s>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Users
>
> > > > > > File">./conf/users.xml</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Legacy Authorized Users
>
> > > > > > File"></property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial User Identity 1">CN=bbende,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=Apache NiFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial User Identity
>
> > > > > > 2">CN=nifi-host-1,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial User Identity
>
> > > > > > 2">CN=nifi-host-2,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > </userGroupProvider>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <accessPolicyProvider>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <identifier>file-access-policy-provider</identifier>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <class>org.apache.nifi.authorization.FileAccessPolicyProvide
> > > > > > r<
>
> > > > > > /c
>
> > > > > > la
>
> > >
>
> > > > > > ss
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="User Group
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Provider">composite-configurable-user-group-provider</proper
> > > > > > ty
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Authorizations
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > File">./conf/authorizations.xml</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial Admin Identity">CN=bbende,
>
> > >
>
> > > > > > OU=Apache
>
> > >
>
> > > >
>
> > >
>
> > > > > > NiFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Legacy Authorized Users
>
> > > > > > File"></property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Node Identity 1">CN=nifi-host-1,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Node Identity 1">CN=nifi-host-2,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > </accessPolicyProvider>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Hi Bryan,
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Are you saying that we have to run ZooKeeper on both nodes?
>
> > > > > > > BTW,
>
> > >
>
> > > >
>
> > >
>
> > > > > > > do
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > we still need
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > nifi.login.identity.provider.configuration.file=./conf/log
> > > > > > > in
>
> > > > > > > -i
>
> > > > > > > de
>
> > >
>
> > > > > > > nt
>
> > >
>
> > > >
>
> > >
>
> > > > > > > it
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > y-
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > providers.xml
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Thank you,
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Alexander
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > -----Original Message-----
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > From: Bryan Bende <bb...@gmail.com>>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Sent: Monday, October 22, 2018 11:55 AM
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > server.1=nifi-node1-hostname:2888:3888
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > server.2=nifi-node2-hostname:2888:3888
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > > I wonder if anyone has run into the same problem when
>
> > > > > > > > trying
>
> > >
>
> > > > > > > > to
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > > configure composite authentication/authorization (LDAP
>
> > > > > > > > and
>
> > >
>
> > > >
>
> > >
>
> > > > > > > > local
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
Many services can share a single ZooKeeper by segmenting their data
under a specific root node.
The root node is specified by nifi.zookeeper.root.node=/nifi so if
those were different on each node then it would form separate
clusters.
Can you show screenshots of the cluster information from each node?
May need to upload them somewhere and provide links here since
attachments don't always make it through.
On Wed, Oct 24, 2018 at 8:18 AM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> The ZooKeeper related settings in the nifi.properties files on both hosts are identical, with the exception of nifi.state.management.embedded.zookeeper.start, which is ‘true’ on host-1 and ‘false’ on host-2. Moreover, if I shut down NiFi on host-1, it crashes on host-2. Here is the message in the browser window:
>
>
>
> Action cannot be performed because there is currently no Cluster Coordinator elected. The request should be tried again after a moment, after a Cluster Coordinator has been automatically elected.
>
>
>
> I even went as far as commenting out the server.1 line in the zookeeper.properties file on host-1 before restarting both NiFi instances, which didn’t change the outcome.
>
>
>
> When I look at the NiFi Cluster information in the UI on host-1, it shows the status of the node “CONNECTED, PRIMARY, COORDINATOR”, whereas on host-2 just “CONNECTED”. I don’t know if this tells you anything.
>
>
>
> BTW, what does “a different location in the same ZK” mean?
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Tuesday, October 23, 2018 3:02 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> The only way I could see that happening is if the ZK config on the second node pointed at a different ZK, or at a different location in the same ZK.
>
>
>
> For example, if node 1 had:
>
>
>
> nifi.zookeeper.connect.string=node-1:2181
>
> nifi.zookeeper.connect.timeout=3 secs
>
> nifi.zookeeper.session.timeout=3 secs
>
> nifi.zookeeper.root.node=/nifi
>
>
>
> Then node 2 should have exactly the same thing.
>
>
>
> If node 2 specified a different connect string, or a different root node, then it wouldn't know about the other node.
>
>
>
> On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > That's exactly the case.
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Tuesday, October 23, 2018 2:44 PM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> >
>
> > It doesn't really make sense how the second node would form its own cluster.
>
> > On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
> > >
>
> > >
>
> > >
>
> > > 2018-10-23 13:44:43,628 INFO
>
> > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
>
> > > o.a.zookeeper.server.ZooKeeperServer Client attempting to establish
>
> > > new session at /<host-2 IP address>:50412
>
> > >
>
> > > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
>
> > > o.a.zookeeper.server.ZooKeeperServer Established session
>
> > > 0x166a1d139590002 with negotiated timeout 4000 for client /<host-2
>
> > > IP
>
> > > address>:50412
>
> > >
>
> > >
>
> > >
>
> > > I apologize for bugging you with all this, converting our standalone
>
> > > NiFi instances into cluster nodes turned out to be much more
>
> > > challenging than we had anticipated…
>
> > >
>
> > >
>
> > >
>
> > > -----Original Message-----
>
> > > From: Bryan Bende <bb...@gmail.com>
>
> > > Sent: Tuesday, October 23, 2018 1:17 PM
>
> > > To: users@nifi.apache.org
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > >
>
> > >
>
> > > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> > >
>
> > > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > -----Original Message-----
>
> > >
>
> > > > From: Bryan Bende <bb...@gmail.com>
>
> > >
>
> > > > Sent: Tuesday, October 23, 2018 12:36 PM
>
> > >
>
> > > > To: users@nifi.apache.org
>
> > >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > That means the user representing host-1 does not have permissions to proxy.
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > You can look in authorizations.xml on nifi-1 for a policy like:
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> > >
>
> > > >
>
> > >
>
> > > > resource="/proxy" action="W">
>
> > >
>
> > > >
>
> > >
>
> > > > <user
>
> > > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> > >
>
> > > >
>
> > >
>
> > > > </policy>
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > That user identifier should point to a user in users.xml like:
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> > >
>
> > > >
>
> > >
>
> > > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> > >
>
> > > >
>
> > >
>
> > > > Government, C=US"/>
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Hi Bryan,
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · restarted one instance in the standalone mode
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · granted them the “proxy user requests” privileges
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > · restarted the node on host-1
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Insufficient Permissions
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > The nifi-user.log also contains
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> > >
>
> > > >
>
> > >
>
> > > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> > >
>
> > > >
>
> > >
>
> > > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH,
>
> > > > > OU=HHS,
>
> > >
>
> > > >
>
> > >
>
> > > > > O=U.S. Government, C=US
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > From your experience, what the most likely causes for this exception?
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Thank you,
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Alexander
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > -----Original Message-----
>
> > >
>
> > > >
>
> > >
>
> > > > > From: Bryan Bende <bb...@gmail.com>
>
> > >
>
> > > >
>
> > >
>
> > > > > Sent: Monday, October 22, 2018 1:25 PM
>
> > >
>
> > > >
>
> > >
>
> > > > > To: users@nifi.apache.org
>
> > >
>
> > > >
>
> > >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Yes, to further clarify what I meant...
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > - In standalone mode, use the UI to add users for the DN's of
>
> > > > > the
>
> > >
>
> > > >
>
> > >
>
> > > > > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2,
>
> > >
>
> > > > > OU=NIFI)
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > - In the UI, grant those users Write access to "Proxy"
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > - Convert to a cluster and keep your same authorizers.xml,
>
> > >
>
> > > > > users.xml,
>
> > >
>
> > > >
>
> > >
>
> > > > > and authorizations.xml when you setup your cluster, this way all
>
> > >
>
> > > > > your
>
> > >
>
> > > >
>
> > >
>
> > > > > users and policies are already setup and the Initial Admin and
>
> > > > > Node
>
> > >
>
> > > >
>
> > >
>
> > > > > Identities are not needed
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > -----Original Message-----
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > From: Bryan Bende <bb...@gmail.com>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Sent: Monday, October 22, 2018 12:48 PM
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > To: users@nifi.apache.org
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > This blog post shows how to setup a secure 2 node cluster:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-
>
> > > > > > 0-
>
> > > > > > au
>
> > >
>
> > > > > > th
>
> > >
>
> > > >
>
> > >
>
> > > > > > or
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > ization-and-multi-tenancy
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <authorizer>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <identifier>file-provider</identifier>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > > > > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Authorizations
>
> > >
>
> > > >
>
> > >
>
> > > > > > File">./conf/authorizations.xml</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Users File">./conf/users.xml</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial Admin Identity">CN=bbende,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=ApacheNiFi</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Legacy Authorized Users File"></property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Node Identity 1">CN=localhost,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > </authorizer>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <userGroupProvider>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <identifier>file-user-group-provider</identifier>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <class>org.apache.nifi.authorization.FileUserGroupProvider</cl
>
> > > > > > as
>
> > > > > > s>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Users
>
> > > > > > File">./conf/users.xml</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Legacy Authorized Users
>
> > > > > > File"></property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial User Identity 1">CN=bbende,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=Apache NiFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial User Identity
>
> > > > > > 2">CN=nifi-host-1,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial User Identity
>
> > > > > > 2">CN=nifi-host-2,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > </userGroupProvider>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <accessPolicyProvider>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <identifier>file-access-policy-provider</identifier>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <class>org.apache.nifi.authorization.FileAccessPolicyProvider<
>
> > > > > > /c
>
> > > > > > la
>
> > >
>
> > > > > > ss
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="User Group
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Provider">composite-configurable-user-group-provider</property
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Authorizations
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > File">./conf/authorizations.xml</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Initial Admin Identity">CN=bbende,
>
> > >
>
> > > > > > OU=Apache
>
> > >
>
> > > >
>
> > >
>
> > > > > > NiFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Legacy Authorized Users
>
> > > > > > File"></property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Node Identity 1">CN=nifi-host-1,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > <property name="Node Identity 1">CN=nifi-host-2,
>
> > >
>
> > > >
>
> > >
>
> > > > > > OU=NIFI</property>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > </accessPolicyProvider>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Hi Bryan,
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Are you saying that we have to run ZooKeeper on both nodes?
>
> > > > > > > BTW,
>
> > >
>
> > > >
>
> > >
>
> > > > > > > do
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > we still need
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > nifi.login.identity.provider.configuration.file=./conf/login
>
> > > > > > > -i
>
> > > > > > > de
>
> > >
>
> > > > > > > nt
>
> > >
>
> > > >
>
> > >
>
> > > > > > > it
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > y-
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > providers.xml
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Thank you,
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Alexander
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > -----Original Message-----
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > From: Bryan Bende <bb...@gmail.com>
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Sent: Monday, October 22, 2018 11:55 AM
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > To: users@nifi.apache.org
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > server.1=nifi-node1-hostname:2888:3888
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > server.2=nifi-node2-hostname:2888:3888
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > > I wonder if anyone has run into the same problem when
>
> > > > > > > > trying
>
> > >
>
> > > > > > > > to
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > > > configure composite authentication/authorization (LDAP
>
> > > > > > > > and
>
> > >
>
> > > >
>
> > >
>
> > > > > > > > local
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
The ZooKeeper related settings in the nifi.properties files on both hosts are identical, with the exception of nifi.state.management.embedded.zookeeper.start, which is ‘true’ on host-1 and ‘false’ on host-2. Moreover, if I shut down NiFi on host-1, it crashes on host-2. Here is the message in the browser window:
Action cannot be performed because there is currently no Cluster Coordinator elected. The request should be tried again after a moment, after a Cluster Coordinator has been automatically elected.
I even went as far as commenting out the server.1 line in the zookeeper.properties file on host-1 before restarting both NiFi instances, which didn’t change the outcome.
When I look at the NiFi Cluster information in the UI on host-1, it shows the status of the node “CONNECTED, PRIMARY, COORDINATOR”, whereas on host-2 just “CONNECTED”. I don’t know if this tells you anything.
BTW, what does “a different location in the same ZK” mean?
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Tuesday, October 23, 2018 3:02 PM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
The only way I could see that happening is if the ZK config on the second node pointed at a different ZK, or at a different location in the same ZK.
For example, if node 1 had:
nifi.zookeeper.connect.string=node-1:2181
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi
Then node 2 should have exactly the same thing.
If node 2 specified a different connect string, or a different root node, then it wouldn't know about the other node.
On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> That's exactly the case.
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Tuesday, October 23, 2018 2:44 PM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
> So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> It doesn't really make sense how the second node would form its own cluster.
> On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
> >
> >
> >
> > 2018-10-23 13:44:43,628 INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> > o.a.zookeeper.server.ZooKeeperServer Client attempting to establish
> > new session at /<host-2 IP address>:50412
> >
> > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
> > o.a.zookeeper.server.ZooKeeperServer Established session
> > 0x166a1d139590002 with negotiated timeout 4000 for client /<host-2
> > IP
> > address>:50412
> >
> >
> >
> > I apologize for bugging you with all this, converting our standalone
> > NiFi instances into cluster nodes turned out to be much more
> > challenging than we had anticipated…
> >
> >
> >
> > -----Original Message-----
> > From: Bryan Bende <bb...@gmail.com>>
> > Sent: Tuesday, October 23, 2018 1:17 PM
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
> > Subject: Re: NiFi fails on cluster nodes
> >
> >
> >
> > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
> >
> > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
> >
> > >
> >
> > >
> >
> > >
> >
> > > -----Original Message-----
> >
> > > From: Bryan Bende <bb...@gmail.com>>
> >
> > > Sent: Tuesday, October 23, 2018 12:36 PM
> >
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > >
> >
> > >
> >
> > > That means the user representing host-1 does not have permissions to proxy.
> >
> > >
> >
> > >
> >
> > >
> >
> > > You can look in authorizations.xml on nifi-1 for a policy like:
> >
> > >
> >
> > >
> >
> > >
> >
> > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
> >
> > >
> >
> > > resource="/proxy" action="W">
> >
> > >
> >
> > > <user
> > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
> >
> > >
> >
> > > </policy>
> >
> > >
> >
> > >
> >
> > >
> >
> > > That user identifier should point to a user in users.xml like:
> >
> > >
> >
> > >
> >
> > >
> >
> > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
> >
> > >
> >
> > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
> >
> > >
> >
> > > Government, C=US"/>
> >
> > >
> >
> > >
> >
> > >
> >
> > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
> >
> > >
> >
> > >
> >
> > >
> >
> > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Hi Bryan,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · restarted one instance in the standalone mode
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · granted them the “proxy user requests” privileges
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · restarted the node on host-1
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Insufficient Permissions
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > The nifi-user.log also contains
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
> >
> > >
> >
> > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
> >
> > >
> >
> > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH,
> > > > OU=HHS,
> >
> > >
> >
> > > > O=U.S. Government, C=US
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > From your experience, what the most likely causes for this exception?
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Thank you,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Alexander
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > -----Original Message-----
> >
> > >
> >
> > > > From: Bryan Bende <bb...@gmail.com>>
> >
> > >
> >
> > > > Sent: Monday, October 22, 2018 1:25 PM
> >
> > >
> >
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Yes, to further clarify what I meant...
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > - In standalone mode, use the UI to add users for the DN's of
> > > > the
> >
> > >
> >
> > > > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2,
> >
> > > > OU=NIFI)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > - In the UI, grant those users Write access to "Proxy"
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > - Convert to a cluster and keep your same authorizers.xml,
> >
> > > > users.xml,
> >
> > >
> >
> > > > and authorizations.xml when you setup your cluster, this way all
> >
> > > > your
> >
> > >
> >
> > > > users and policies are already setup and the Initial Admin and
> > > > Node
> >
> > >
> >
> > > > Identities are not needed
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > -----Original Message-----
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > From: Bryan Bende <bb...@gmail.com>>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Sent: Monday, October 22, 2018 12:48 PM
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > This blog post shows how to setup a secure 2 node cluster:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-
> > > > > 0-
> > > > > au
> >
> > > > > th
> >
> > >
> >
> > > > > or
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > ization-and-multi-tenancy
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <authorizer>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <identifier>file-provider</identifier>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> > > > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Authorizations
> >
> > >
> >
> > > > > File">./conf/authorizations.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Users File">./conf/users.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial Admin Identity">CN=bbende,
> >
> > >
> >
> > > > > OU=ApacheNiFi</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Legacy Authorized Users File"></property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Node Identity 1">CN=localhost,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > </authorizer>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > You need to add the the users to the user-group-provider and then to the access-policy-provider...
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <userGroupProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <identifier>file-user-group-provider</identifier>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > <class>org.apache.nifi.authorization.FileUserGroupProvider</cl
> > > > > as
> > > > > s>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Users
> > > > > File">./conf/users.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Legacy Authorized Users
> > > > > File"></property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial User Identity 1">CN=bbende,
> >
> > >
> >
> > > > > OU=Apache NiFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial User Identity
> > > > > 2">CN=nifi-host-1,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial User Identity
> > > > > 2">CN=nifi-host-2,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > </userGroupProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <accessPolicyProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <identifier>file-access-policy-provider</identifier>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > <class>org.apache.nifi.authorization.FileAccessPolicyProvider<
> > > > > /c
> > > > > la
> >
> > > > > ss
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="User Group
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Provider">composite-configurable-user-group-provider</property
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Authorizations
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > File">./conf/authorizations.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial Admin Identity">CN=bbende,
> >
> > > > > OU=Apache
> >
> > >
> >
> > > > > NiFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Legacy Authorized Users
> > > > > File"></property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Node Identity 1">CN=nifi-host-1,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Node Identity 1">CN=nifi-host-2,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > </accessPolicyProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Hi Bryan,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Are you saying that we have to run ZooKeeper on both nodes?
> > > > > > BTW,
> >
> > >
> >
> > > > > > do
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > we still need
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > nifi.login.identity.provider.configuration.file=./conf/login
> > > > > > -i
> > > > > > de
> >
> > > > > > nt
> >
> > >
> >
> > > > > > it
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > y-
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > providers.xml
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Thank you,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Alexander
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > -----Original Message-----
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > From: Bryan Bende <bb...@gmail.com>>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Sent: Monday, October 22, 2018 11:55 AM
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > server.1=nifi-node1-hostname:2888:3888
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > server.2=nifi-node2-hostname:2888:3888
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > I wonder if anyone has run into the same problem when
> > > > > > > trying
> >
> > > > > > > to
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > configure composite authentication/authorization (LDAP
> > > > > > > and
> >
> > >
> >
> > > > > > > local
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > file)? When we use the “stand-alone” authorizers.xml file
> > > > > > > with
> >
> > >
> >
> > > > > > > the
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > addition of two extra properties
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > <property name="Node Identity 1">…
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > <property name="Node Identity 2">…
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > and let ZooKeeper start on one on the nodes, we end up
> > > > > > > with
> >
> > > > > > > two
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > one-node clusters, since apparently, the NiFi instances
> > > > > > > don’t
> >
> > >
> >
> > > > > > > talk
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > to
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > each other, but at least, they come alive…
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > From: Saip, Alexander (NIH/CC/BTRIS) [C]
> >
> > >
> >
> > > > > > > <al...@nih.gov>>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Sent: Friday, October 19, 2018 11:18 AM
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Subject: RE: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 2018-10-19 08:09:26,992 ERROR [main]
> >
> > >
> >
> > > > > > > o.s.web.context.ContextLoader
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Context initialization failed
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Error creating bean with name
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 'org.springframework.security.config.annotation.web.configuration.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > We
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > bS
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ecurityConfiguration': Unsatisfied dependency expressed
> >
> > > > > > > through
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > method
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 'setFilterChainProxySecurityConfigurer' parameter 1;
> > > > > > > nested
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > exception
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > is org.springframework.beans.factory.BeanExpressionException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Expression parsing failed; nested exception is
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Error creating bean with name
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
> >
> > >
> >
> > > > > > > Unsatisfied
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > dependency expressed through method 'setJwtAuthenticationProvider'
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > parameter 0; nested exception is
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.BeanCreationException:
> > > > > > > Error
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > creating bean with name 'jwtAuthenticationProvider'
> > > > > > > defined in
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > class
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > path resource [nifi-web-security-context.xml]: Cannot
> > > > > > > resolve
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > reference to bean 'authorizer' while setting constructor
> >
> > >
> >
> > > > > > > argument;
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nested exception is
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.BeanCreationException:
> > > > > > > Error
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > creating bean with name 'authorizer': FactoryBean threw
> >
> > >
> >
> > > > > > > exception
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > on
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > object creation; nested exception is java.lang.NullPointerException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Name is null
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.annotation.AutowiredAnno
> > > > > > > ta
> > > > > > > ti
> >
> > > > > > > on
> >
> > >
> >
> > > > > > > Be
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > an
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Po
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotat
> > > > > > > io
> > > > > > > nB
> >
> > > > > > > ea
> >
> > >
> >
> > > > > > > nP
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > os
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > tP
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > rocessor.java:667)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.annotation.InjectionMeta
> > > > > > > da
> > > > > > > ta
> >
> > > > > > > .i
> >
> > >
> >
> > > > > > > nj
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ec
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > t(
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > InjectionMetadata.java:88)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.annotation.AutowiredAnno
> > > > > > > ta
> > > > > > > ti
> >
> > > > > > > on
> >
> > >
> >
> > > > > > > Be
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > an
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Po
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationB
> > > > > > > ea
> > > > > > > nP
> >
> > > > > > > os
> >
> > >
> >
> > > > > > > tP
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ro
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ce
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ssor.java:366)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractAutowire
> > > > > > > Ca
> > > > > > > pa
> >
> > > > > > > bl
> >
> > >
> >
> > > > > > > eB
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ea
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nF
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:
> > > > > > > 12
> >
> > > > > > > 64
> >
> > >
> >
> > > > > > > )
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractAutowire
> > > > > > > Ca
> > > > > > > pa
> >
> > > > > > > bl
> >
> > >
> >
> > > > > > > eB
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ea
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nF
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:
> > > > > > > 55
> >
> > > > > > > 3)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractAutowire
> > > > > > > Ca
> > > > > > > pa
> >
> > > > > > > bl
> >
> > >
> >
> > > > > > > eB
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ea
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nF
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:
> > > > > > > 48
> > > > > > > 3)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.
> >
> > >
> >
> > > > > > > ge
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > tO
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > bj
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ect(AbstractBeanFactory.java:306)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.DefaultSingleton
> > > > > > > Be
> > > > > > > an
> >
> > > > > > > Re
> >
> > >
> >
> > > > > > > gi
> >
> > >
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
The only way I could see that happening is if the ZK config on the
second node pointed at a different ZK, or at a different location in
the same ZK.
For example, if node 1 had:
nifi.zookeeper.connect.string=node-1:2181
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi
Then node 2 should have exactly the same thing.
If node 2 specified a different connect string, or a different root
node, then it wouldn't know about the other node.
On Tue, Oct 23, 2018 at 2:48 PM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> That's exactly the case.
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Tuesday, October 23, 2018 2:44 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
> So you can get into each node's UI and they each show 1/1 for cluster nodes?
>
> It doesn't really make sense how the second node would form its own cluster.
> On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
> >
> >
> >
> > 2018-10-23 13:44:43,628 INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> > o.a.zookeeper.server.ZooKeeperServer Client attempting to establish
> > new session at /<host-2 IP address>:50412
> >
> > 2018-10-23 13:44:43,629 INFO [SyncThread:0]
> > o.a.zookeeper.server.ZooKeeperServer Established session
> > 0x166a1d139590002 with negotiated timeout 4000 for client /<host-2 IP
> > address>:50412
> >
> >
> >
> > I apologize for bugging you with all this, converting our standalone
> > NiFi instances into cluster nodes turned out to be much more
> > challenging than we had anticipated…
> >
> >
> >
> > -----Original Message-----
> > From: Bryan Bende <bb...@gmail.com>
> > Sent: Tuesday, October 23, 2018 1:17 PM
> > To: users@nifi.apache.org
> > Subject: Re: NiFi fails on cluster nodes
> >
> >
> >
> > Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
> >
> > On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
> >
> > >
> >
> > >
> >
> > >
> >
> > > -----Original Message-----
> >
> > > From: Bryan Bende <bb...@gmail.com>
> >
> > > Sent: Tuesday, October 23, 2018 12:36 PM
> >
> > > To: users@nifi.apache.org
> >
> > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > >
> >
> > >
> >
> > > That means the user representing host-1 does not have permissions to proxy.
> >
> > >
> >
> > >
> >
> > >
> >
> > > You can look in authorizations.xml on nifi-1 for a policy like:
> >
> > >
> >
> > >
> >
> > >
> >
> > > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
> >
> > >
> >
> > > resource="/proxy" action="W">
> >
> > >
> >
> > > <user
> > > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
> >
> > >
> >
> > > </policy>
> >
> > >
> >
> > >
> >
> > >
> >
> > > That user identifier should point to a user in users.xml like:
> >
> > >
> >
> > >
> >
> > >
> >
> > > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
> >
> > >
> >
> > > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
> >
> > >
> >
> > > Government, C=US"/>
> >
> > >
> >
> > >
> >
> > >
> >
> > > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
> >
> > >
> >
> > >
> >
> > >
> >
> > > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Hi Bryan,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · restarted one instance in the standalone mode
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · granted them the “proxy user requests” privileges
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > · restarted the node on host-1
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Insufficient Permissions
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > The nifi-user.log also contains
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
> >
> > >
> >
> > > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
> >
> > >
> >
> > > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS,
> >
> > >
> >
> > > > O=U.S. Government, C=US
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > From your experience, what the most likely causes for this exception?
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Thank you,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Alexander
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > -----Original Message-----
> >
> > >
> >
> > > > From: Bryan Bende <bb...@gmail.com>
> >
> > >
> >
> > > > Sent: Monday, October 22, 2018 1:25 PM
> >
> > >
> >
> > > > To: users@nifi.apache.org
> >
> > >
> >
> > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Yes, to further clarify what I meant...
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > - In standalone mode, use the UI to add users for the DN's of the
> >
> > >
> >
> > > > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2,
> >
> > > > OU=NIFI)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > - In the UI, grant those users Write access to "Proxy"
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > - Convert to a cluster and keep your same authorizers.xml,
> >
> > > > users.xml,
> >
> > >
> >
> > > > and authorizations.xml when you setup your cluster, this way all
> >
> > > > your
> >
> > >
> >
> > > > users and policies are already setup and the Initial Admin and
> > > > Node
> >
> > >
> >
> > > > Identities are not needed
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > -----Original Message-----
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > From: Bryan Bende <bb...@gmail.com>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Sent: Monday, October 22, 2018 12:48 PM
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > To: users@nifi.apache.org
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > This blog post shows how to setup a secure 2 node cluster:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-
> > > > > au
> >
> > > > > th
> >
> > >
> >
> > > > > or
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > ization-and-multi-tenancy
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <authorizer>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <identifier>file-provider</identifier>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Authorizations
> >
> > >
> >
> > > > > File">./conf/authorizations.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Users File">./conf/users.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial Admin Identity">CN=bbende,
> >
> > >
> >
> > > > > OU=ApacheNiFi</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Legacy Authorized Users File"></property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Node Identity 1">CN=localhost,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > </authorizer>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > You need to add the the users to the user-group-provider and then to the access-policy-provider...
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <userGroupProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <identifier>file-user-group-provider</identifier>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > <class>org.apache.nifi.authorization.FileUserGroupProvider</clas
> > > > > s>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Users File">./conf/users.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Legacy Authorized Users
> > > > > File"></property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial User Identity 1">CN=bbende,
> >
> > >
> >
> > > > > OU=Apache NiFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial User Identity 2">CN=nifi-host-1,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial User Identity 2">CN=nifi-host-2,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > </userGroupProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <accessPolicyProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <identifier>file-access-policy-provider</identifier>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</c
> > > > > la
> >
> > > > > ss
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="User Group
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Provider">composite-configurable-user-group-provider</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Authorizations
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > File">./conf/authorizations.xml</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Initial Admin Identity">CN=bbende,
> >
> > > > > OU=Apache
> >
> > >
> >
> > > > > NiFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Legacy Authorized Users
> > > > > File"></property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Node Identity 1">CN=nifi-host-1,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > <property name="Node Identity 1">CN=nifi-host-2,
> >
> > >
> >
> > > > > OU=NIFI</property>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > </accessPolicyProvider>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Hi Bryan,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Are you saying that we have to run ZooKeeper on both nodes?
> > > > > > BTW,
> >
> > >
> >
> > > > > > do
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > we still need
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > nifi.login.identity.provider.configuration.file=./conf/login-i
> > > > > > de
> >
> > > > > > nt
> >
> > >
> >
> > > > > > it
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > y-
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > providers.xml
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Thank you,
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Alexander
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > -----Original Message-----
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > From: Bryan Bende <bb...@gmail.com>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Sent: Monday, October 22, 2018 11:55 AM
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > To: users@nifi.apache.org
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > server.1=nifi-node1-hostname:2888:3888
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > server.2=nifi-node2-hostname:2888:3888
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > I wonder if anyone has run into the same problem when trying
> >
> > > > > > > to
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > configure composite authentication/authorization (LDAP and
> >
> > >
> >
> > > > > > > local
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > file)? When we use the “stand-alone” authorizers.xml file
> > > > > > > with
> >
> > >
> >
> > > > > > > the
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > addition of two extra properties
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > <property name="Node Identity 1">…
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > <property name="Node Identity 2">…
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > and let ZooKeeper start on one on the nodes, we end up with
> >
> > > > > > > two
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > one-node clusters, since apparently, the NiFi instances
> > > > > > > don’t
> >
> > >
> >
> > > > > > > talk
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > to
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > each other, but at least, they come alive…
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > From: Saip, Alexander (NIH/CC/BTRIS) [C]
> >
> > >
> >
> > > > > > > <al...@nih.gov>
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Sent: Friday, October 19, 2018 11:18 AM
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > To: users@nifi.apache.org
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Subject: RE: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 2018-10-19 08:09:26,992 ERROR [main]
> >
> > >
> >
> > > > > > > o.s.web.context.ContextLoader
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Context initialization failed
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Error creating bean with name
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 'org.springframework.security.config.annotation.web.configuration.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > We
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > bS
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ecurityConfiguration': Unsatisfied dependency expressed
> >
> > > > > > > through
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > method
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > exception
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > is org.springframework.beans.factory.BeanExpressionException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Expression parsing failed; nested exception is
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Error creating bean with name
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
> >
> > >
> >
> > > > > > > Unsatisfied
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > dependency expressed through method 'setJwtAuthenticationProvider'
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > parameter 0; nested exception is
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.BeanCreationException:
> > > > > > > Error
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > creating bean with name 'jwtAuthenticationProvider' defined
> > > > > > > in
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > class
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > path resource [nifi-web-security-context.xml]: Cannot
> > > > > > > resolve
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > reference to bean 'authorizer' while setting constructor
> >
> > >
> >
> > > > > > > argument;
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nested exception is
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.BeanCreationException:
> > > > > > > Error
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > creating bean with name 'authorizer': FactoryBean threw
> >
> > >
> >
> > > > > > > exception
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > on
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > object creation; nested exception is java.lang.NullPointerException:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Name is null
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.annotation.AutowiredAnnota
> > > > > > > ti
> >
> > > > > > > on
> >
> > >
> >
> > > > > > > Be
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > an
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Po
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotatio
> > > > > > > nB
> >
> > > > > > > ea
> >
> > >
> >
> > > > > > > nP
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > os
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > tP
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > rocessor.java:667)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.annotation.InjectionMetada
> > > > > > > ta
> >
> > > > > > > .i
> >
> > >
> >
> > > > > > > nj
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ec
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > t(
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > InjectionMetadata.java:88)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.annotation.AutowiredAnnota
> > > > > > > ti
> >
> > > > > > > on
> >
> > >
> >
> > > > > > > Be
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > an
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > Po
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBea
> > > > > > > nP
> >
> > > > > > > os
> >
> > >
> >
> > > > > > > tP
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ro
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ce
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ssor.java:366)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractAutowireCa
> > > > > > > pa
> >
> > > > > > > bl
> >
> > >
> >
> > > > > > > eB
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ea
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nF
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:
> > > > > > > 12
> >
> > > > > > > 64
> >
> > >
> >
> > > > > > > )
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractAutowireCa
> > > > > > > pa
> >
> > > > > > > bl
> >
> > >
> >
> > > > > > > eB
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ea
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nF
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:
> > > > > > > 55
> >
> > > > > > > 3)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractAutowireCa
> > > > > > > pa
> >
> > > > > > > bl
> >
> > >
> >
> > > > > > > eB
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ea
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > nF
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:48
> > > > > > > 3)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.
> >
> > >
> >
> > > > > > > ge
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > tO
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > bj
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > ect(AbstractBeanFactory.java:306)
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > at
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > >
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > > > > org.springframework.beans.factory.support.DefaultSingletonBe
> > > > > > > an
> >
> > > > > > > Re
> >
> > >
> >
> > > > > > > gi
> >
> > >
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
That's exactly the case.
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Tuesday, October 23, 2018 2:44 PM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
So you can get into each node's UI and they each show 1/1 for cluster nodes?
It doesn't really make sense how the second node would form its own cluster.
On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
>
>
> 2018-10-23 13:44:43,628 INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> o.a.zookeeper.server.ZooKeeperServer Client attempting to establish
> new session at /<host-2 IP address>:50412
>
> 2018-10-23 13:44:43,629 INFO [SyncThread:0]
> o.a.zookeeper.server.ZooKeeperServer Established session
> 0x166a1d139590002 with negotiated timeout 4000 for client /<host-2 IP
> address>:50412
>
>
>
> I apologize for bugging you with all this, converting our standalone
> NiFi instances into cluster nodes turned out to be much more
> challenging than we had anticipated…
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Tuesday, October 23, 2018 1:17 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Tuesday, October 23, 2018 12:36 PM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > That means the user representing host-1 does not have permissions to proxy.
>
> >
>
> >
>
> >
>
> > You can look in authorizations.xml on nifi-1 for a policy like:
>
> >
>
> >
>
> >
>
> > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> >
>
> > resource="/proxy" action="W">
>
> >
>
> > <user
> > identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> >
>
> > </policy>
>
> >
>
> >
>
> >
>
> > That user identifier should point to a user in users.xml like:
>
> >
>
> >
>
> >
>
> > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> >
>
> > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> >
>
> > Government, C=US"/>
>
> >
>
> >
>
> >
>
> > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> >
>
> >
>
> >
>
> > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > Hi Bryan,
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > >
>
> >
>
> > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > >
>
> >
>
> > > · restarted one instance in the standalone mode
>
> >
>
> > >
>
> >
>
> > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > >
>
> >
>
> > > · granted them the “proxy user requests” privileges
>
> >
>
> > >
>
> >
>
> > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > >
>
> >
>
> > > · restarted the node on host-1
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Insufficient Permissions
>
> >
>
> > >
>
> >
>
> > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > The nifi-user.log also contains
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> >
>
> > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> >
>
> > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS,
>
> >
>
> > > O=U.S. Government, C=US
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > From your experience, what the most likely causes for this exception?
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Thank you,
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Alexander
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > Sent: Monday, October 22, 2018 1:25 PM
>
> >
>
> > > To: users@nifi.apache.org
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Yes, to further clarify what I meant...
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > - In standalone mode, use the UI to add users for the DN's of the
>
> >
>
> > > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2,
>
> > > OU=NIFI)
>
> >
>
> > >
>
> >
>
> > > - In the UI, grant those users Write access to "Proxy"
>
> >
>
> > >
>
> >
>
> > > - Convert to a cluster and keep your same authorizers.xml,
>
> > > users.xml,
>
> >
>
> > > and authorizations.xml when you setup your cluster, this way all
>
> > > your
>
> >
>
> > > users and policies are already setup and the Initial Admin and
> > > Node
>
> >
>
> > > Identities are not needed
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > >
>
> >
>
> > > > Sent: Monday, October 22, 2018 12:48 PM
>
> >
>
> > >
>
> >
>
> > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > This blog post shows how to setup a secure 2 node cluster:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-
> > > > au
>
> > > > th
>
> >
>
> > > > or
>
> >
>
> > >
>
> >
>
> > > > ization-and-multi-tenancy
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > <authorizer>
>
> >
>
> > >
>
> >
>
> > > > <identifier>file-provider</identifier>
>
> >
>
> > >
>
> >
>
> > > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
>
> >
>
> > >
>
> >
>
> > > > <property name="Authorizations
>
> >
>
> > > > File">./conf/authorizations.xml</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Users File">./conf/users.xml</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial Admin Identity">CN=bbende,
>
> >
>
> > > > OU=ApacheNiFi</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Node Identity 1">CN=localhost,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > </authorizer>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > <userGroupProvider>
>
> >
>
> > >
>
> >
>
> > > > <identifier>file-user-group-provider</identifier>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > > > <class>org.apache.nifi.authorization.FileUserGroupProvider</clas
> > > > s>
>
> >
>
> > >
>
> >
>
> > > > <property name="Users File">./conf/users.xml</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Legacy Authorized Users
> > > > File"></property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial User Identity 1">CN=bbende,
>
> >
>
> > > > OU=Apache NiFI</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial User Identity 2">CN=nifi-host-1,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial User Identity 2">CN=nifi-host-2,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > </userGroupProvider>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > <accessPolicyProvider>
>
> >
>
> > >
>
> >
>
> > > > <identifier>file-access-policy-provider</identifier>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > > > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</c
> > > > la
>
> > > > ss
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > <property name="User Group
>
> >
>
> > >
>
> >
>
> > > > Provider">composite-configurable-user-group-provider</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Authorizations
>
> >
>
> > >
>
> >
>
> > > > File">./conf/authorizations.xml</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial Admin Identity">CN=bbende,
>
> > > > OU=Apache
>
> >
>
> > > > NiFI</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Legacy Authorized Users
> > > > File"></property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Node Identity 1">CN=nifi-host-1,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Node Identity 1">CN=nifi-host-2,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > </accessPolicyProvider>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Hi Bryan,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Are you saying that we have to run ZooKeeper on both nodes?
> > > > > BTW,
>
> >
>
> > > > > do
>
> >
>
> > >
>
> >
>
> > > > > we still need
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > nifi.login.identity.provider.configuration.file=./conf/login-i
> > > > > de
>
> > > > > nt
>
> >
>
> > > > > it
>
> >
>
> > >
>
> >
>
> > > > > y-
>
> >
>
> > >
>
> >
>
> > > > > providers.xml
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Thank you,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Alexander
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > >
>
> >
>
> > > > > Sent: Monday, October 22, 2018 11:55 AM
>
> >
>
> > >
>
> >
>
> > > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > server.1=nifi-node1-hostname:2888:3888
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > server.2=nifi-node2-hostname:2888:3888
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > I wonder if anyone has run into the same problem when trying
>
> > > > > > to
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > configure composite authentication/authorization (LDAP and
>
> >
>
> > > > > > local
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > file)? When we use the “stand-alone” authorizers.xml file
> > > > > > with
>
> >
>
> > > > > > the
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > addition of two extra properties
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <property name="Node Identity 1">…
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <property name="Node Identity 2">…
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > and let ZooKeeper start on one on the nodes, we end up with
>
> > > > > > two
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > one-node clusters, since apparently, the NiFi instances
> > > > > > don’t
>
> >
>
> > > > > > talk
>
> >
>
> > >
>
> >
>
> > > > > > to
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > each other, but at least, they come alive…
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > From: Saip, Alexander (NIH/CC/BTRIS) [C]
>
> >
>
> > > > > > <al...@nih.gov>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Sent: Friday, October 19, 2018 11:18 AM
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Subject: RE: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > 2018-10-19 08:09:26,992 ERROR [main]
>
> >
>
> > > > > > o.s.web.context.ContextLoader
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Context initialization failed
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Error creating bean with name
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > 'org.springframework.security.config.annotation.web.configuration.
>
> >
>
> > >
>
> >
>
> > > > > > We
>
> >
>
> > >
>
> >
>
> > > > > > bS
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > ecurityConfiguration': Unsatisfied dependency expressed
>
> > > > > > through
>
> >
>
> > >
>
> >
>
> > > > > > method
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
>
> >
>
> > >
>
> >
>
> > > > > > exception
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > is org.springframework.beans.factory.BeanExpressionException:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Expression parsing failed; nested exception is
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Error creating bean with name
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
>
> >
>
> > > > > > Unsatisfied
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > parameter 0; nested exception is
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.BeanCreationException:
> > > > > > Error
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > creating bean with name 'jwtAuthenticationProvider' defined
> > > > > > in
>
> >
>
> > >
>
> >
>
> > > > > > class
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > path resource [nifi-web-security-context.xml]: Cannot
> > > > > > resolve
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > reference to bean 'authorizer' while setting constructor
>
> >
>
> > > > > > argument;
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > nested exception is
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.BeanCreationException:
> > > > > > Error
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > creating bean with name 'authorizer': FactoryBean threw
>
> >
>
> > > > > > exception
>
> >
>
> > >
>
> >
>
> > > > > > on
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > object creation; nested exception is java.lang.NullPointerException:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Name is null
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.annotation.AutowiredAnnota
> > > > > > ti
>
> > > > > > on
>
> >
>
> > > > > > Be
>
> >
>
> > >
>
> >
>
> > > > > > an
>
> >
>
> > >
>
> >
>
> > > > > > Po
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotatio
> > > > > > nB
>
> > > > > > ea
>
> >
>
> > > > > > nP
>
> >
>
> > >
>
> >
>
> > > > > > os
>
> >
>
> > >
>
> >
>
> > > > > > tP
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > rocessor.java:667)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.annotation.InjectionMetada
> > > > > > ta
>
> > > > > > .i
>
> >
>
> > > > > > nj
>
> >
>
> > >
>
> >
>
> > > > > > ec
>
> >
>
> > >
>
> >
>
> > > > > > t(
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > InjectionMetadata.java:88)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.annotation.AutowiredAnnota
> > > > > > ti
>
> > > > > > on
>
> >
>
> > > > > > Be
>
> >
>
> > >
>
> >
>
> > > > > > an
>
> >
>
> > >
>
> >
>
> > > > > > Po
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBea
> > > > > > nP
>
> > > > > > os
>
> >
>
> > > > > > tP
>
> >
>
> > >
>
> >
>
> > > > > > ro
>
> >
>
> > >
>
> >
>
> > > > > > ce
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > ssor.java:366)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.AbstractAutowireCa
> > > > > > pa
>
> > > > > > bl
>
> >
>
> > > > > > eB
>
> >
>
> > >
>
> >
>
> > > > > > ea
>
> >
>
> > >
>
> >
>
> > > > > > nF
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:
> > > > > > 12
>
> > > > > > 64
>
> >
>
> > > > > > )
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.AbstractAutowireCa
> > > > > > pa
>
> > > > > > bl
>
> >
>
> > > > > > eB
>
> >
>
> > >
>
> >
>
> > > > > > ea
>
> >
>
> > >
>
> >
>
> > > > > > nF
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:
> > > > > > 55
>
> > > > > > 3)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.AbstractAutowireCa
> > > > > > pa
>
> > > > > > bl
>
> >
>
> > > > > > eB
>
> >
>
> > >
>
> >
>
> > > > > > ea
>
> >
>
> > >
>
> >
>
> > > > > > nF
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:48
> > > > > > 3)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.
>
> >
>
> > > > > > ge
>
> >
>
> > >
>
> >
>
> > > > > > tO
>
> >
>
> > >
>
> >
>
> > > > > > bj
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > ect(AbstractBeanFactory.java:306)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.DefaultSingletonBe
> > > > > > an
>
> > > > > > Re
>
> >
>
> > > > > > gi
>
> >
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
So you can get into each node's UI and they each show 1/1 for cluster nodes?
It doesn't really make sense how the second node would form its own cluster.
On Tue, Oct 23, 2018 at 2:20 PM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
>
>
>
> 2018-10-23 13:44:43,628 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Client attempting to establish new session at /<host-2 IP address>:50412
>
> 2018-10-23 13:44:43,629 INFO [SyncThread:0] o.a.zookeeper.server.ZooKeeperServer Established session 0x166a1d139590002 with negotiated timeout 4000 for client /<host-2 IP address>:50412
>
>
>
> I apologize for bugging you with all this, converting our standalone NiFi instances into cluster nodes turned out to be much more challenging than we had anticipated…
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Tuesday, October 23, 2018 1:17 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
>
> On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Tuesday, October 23, 2018 12:36 PM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > That means the user representing host-1 does not have permissions to proxy.
>
> >
>
> >
>
> >
>
> > You can look in authorizations.xml on nifi-1 for a policy like:
>
> >
>
> >
>
> >
>
> > <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> >
>
> > resource="/proxy" action="W">
>
> >
>
> > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> >
>
> > </policy>
>
> >
>
> >
>
> >
>
> > That user identifier should point to a user in users.xml like:
>
> >
>
> >
>
> >
>
> > <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> >
>
> > identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> >
>
> > Government, C=US"/>
>
> >
>
> >
>
> >
>
> > All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
> >
>
> >
>
> >
>
> > On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > Hi Bryan,
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > >
>
> >
>
> > > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > >
>
> >
>
> > > · restarted one instance in the standalone mode
>
> >
>
> > >
>
> >
>
> > > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > >
>
> >
>
> > > · granted them the “proxy user requests” privileges
>
> >
>
> > >
>
> >
>
> > > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > >
>
> >
>
> > > · restarted the node on host-1
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Insufficient Permissions
>
> >
>
> > >
>
> >
>
> > > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > The nifi-user.log also contains
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> >
>
> > > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> >
>
> > > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS,
>
> >
>
> > > O=U.S. Government, C=US
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > From your experience, what the most likely causes for this exception?
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Thank you,
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Alexander
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > Sent: Monday, October 22, 2018 1:25 PM
>
> >
>
> > > To: users@nifi.apache.org
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Yes, to further clarify what I meant...
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > - In standalone mode, use the UI to add users for the DN's of the
>
> >
>
> > > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2,
>
> > > OU=NIFI)
>
> >
>
> > >
>
> >
>
> > > - In the UI, grant those users Write access to "Proxy"
>
> >
>
> > >
>
> >
>
> > > - Convert to a cluster and keep your same authorizers.xml,
>
> > > users.xml,
>
> >
>
> > > and authorizations.xml when you setup your cluster, this way all
>
> > > your
>
> >
>
> > > users and policies are already setup and the Initial Admin and Node
>
> >
>
> > > Identities are not needed
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > >
>
> >
>
> > > > Sent: Monday, October 22, 2018 12:48 PM
>
> >
>
> > >
>
> >
>
> > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > This blog post shows how to setup a secure 2 node cluster:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-au
>
> > > > th
>
> >
>
> > > > or
>
> >
>
> > >
>
> >
>
> > > > ization-and-multi-tenancy
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > <authorizer>
>
> >
>
> > >
>
> >
>
> > > > <identifier>file-provider</identifier>
>
> >
>
> > >
>
> >
>
> > > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
>
> >
>
> > >
>
> >
>
> > > > <property name="Authorizations
>
> >
>
> > > > File">./conf/authorizations.xml</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Users File">./conf/users.xml</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial Admin Identity">CN=bbende,
>
> >
>
> > > > OU=ApacheNiFi</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Node Identity 1">CN=localhost,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > </authorizer>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > <userGroupProvider>
>
> >
>
> > >
>
> >
>
> > > > <identifier>file-user-group-provider</identifier>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > > > <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
>
> >
>
> > >
>
> >
>
> > > > <property name="Users File">./conf/users.xml</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial User Identity 1">CN=bbende,
>
> >
>
> > > > OU=Apache NiFI</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial User Identity 2">CN=nifi-host-1,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial User Identity 2">CN=nifi-host-2,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > </userGroupProvider>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > <accessPolicyProvider>
>
> >
>
> > >
>
> >
>
> > > > <identifier>file-access-policy-provider</identifier>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > > > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</cla
>
> > > > ss
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > <property name="User Group
>
> >
>
> > >
>
> >
>
> > > > Provider">composite-configurable-user-group-provider</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Authorizations
>
> >
>
> > >
>
> >
>
> > > > File">./conf/authorizations.xml</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Initial Admin Identity">CN=bbende,
>
> > > > OU=Apache
>
> >
>
> > > > NiFI</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Node Identity 1">CN=nifi-host-1,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > <property name="Node Identity 1">CN=nifi-host-2,
>
> >
>
> > > > OU=NIFI</property>
>
> >
>
> > >
>
> >
>
> > > > </accessPolicyProvider>
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Hi Bryan,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Are you saying that we have to run ZooKeeper on both nodes? BTW,
>
> >
>
> > > > > do
>
> >
>
> > >
>
> >
>
> > > > > we still need
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > nifi.login.identity.provider.configuration.file=./conf/login-ide
>
> > > > > nt
>
> >
>
> > > > > it
>
> >
>
> > >
>
> >
>
> > > > > y-
>
> >
>
> > >
>
> >
>
> > > > > providers.xml
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Thank you,
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > Alexander
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > -----Original Message-----
>
> >
>
> > >
>
> >
>
> > > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > >
>
> >
>
> > > > > Sent: Monday, October 22, 2018 11:55 AM
>
> >
>
> > >
>
> >
>
> > > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > server.1=nifi-node1-hostname:2888:3888
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > server.2=nifi-node2-hostname:2888:3888
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > I wonder if anyone has run into the same problem when trying
>
> > > > > > to
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > configure composite authentication/authorization (LDAP and
>
> >
>
> > > > > > local
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > file)? When we use the “stand-alone” authorizers.xml file with
>
> >
>
> > > > > > the
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > addition of two extra properties
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <property name="Node Identity 1">…
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > <property name="Node Identity 2">…
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > and let ZooKeeper start on one on the nodes, we end up with
>
> > > > > > two
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > one-node clusters, since apparently, the NiFi instances don’t
>
> >
>
> > > > > > talk
>
> >
>
> > >
>
> >
>
> > > > > > to
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > each other, but at least, they come alive…
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > From: Saip, Alexander (NIH/CC/BTRIS) [C]
>
> >
>
> > > > > > <al...@nih.gov>
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Sent: Friday, October 19, 2018 11:18 AM
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > To: users@nifi.apache.org
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Subject: RE: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > 2018-10-19 08:09:26,992 ERROR [main]
>
> >
>
> > > > > > o.s.web.context.ContextLoader
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Context initialization failed
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Error creating bean with name
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > 'org.springframework.security.config.annotation.web.configuration.
>
> >
>
> > >
>
> >
>
> > > > > > We
>
> >
>
> > >
>
> >
>
> > > > > > bS
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > ecurityConfiguration': Unsatisfied dependency expressed
>
> > > > > > through
>
> >
>
> > >
>
> >
>
> > > > > > method
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
>
> >
>
> > >
>
> >
>
> > > > > > exception
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > is org.springframework.beans.factory.BeanExpressionException:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Expression parsing failed; nested exception is
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Error creating bean with name
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
>
> >
>
> > > > > > Unsatisfied
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > parameter 0; nested exception is
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.BeanCreationException: Error
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > creating bean with name 'jwtAuthenticationProvider' defined in
>
> >
>
> > >
>
> >
>
> > > > > > class
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > reference to bean 'authorizer' while setting constructor
>
> >
>
> > > > > > argument;
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > nested exception is
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.BeanCreationException: Error
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > creating bean with name 'authorizer': FactoryBean threw
>
> >
>
> > > > > > exception
>
> >
>
> > >
>
> >
>
> > > > > > on
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > object creation; nested exception is java.lang.NullPointerException:
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > Name is null
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotati
>
> > > > > > on
>
> >
>
> > > > > > Be
>
> >
>
> > >
>
> >
>
> > > > > > an
>
> >
>
> > >
>
> >
>
> > > > > > Po
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationB
>
> > > > > > ea
>
> >
>
> > > > > > nP
>
> >
>
> > >
>
> >
>
> > > > > > os
>
> >
>
> > >
>
> >
>
> > > > > > tP
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > rocessor.java:667)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.annotation.InjectionMetadata
>
> > > > > > .i
>
> >
>
> > > > > > nj
>
> >
>
> > >
>
> >
>
> > > > > > ec
>
> >
>
> > >
>
> >
>
> > > > > > t(
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > InjectionMetadata.java:88)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.annotation.AutowiredAnnotati
>
> > > > > > on
>
> >
>
> > > > > > Be
>
> >
>
> > >
>
> >
>
> > > > > > an
>
> >
>
> > >
>
> >
>
> > > > > > Po
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanP
>
> > > > > > os
>
> >
>
> > > > > > tP
>
> >
>
> > >
>
> >
>
> > > > > > ro
>
> >
>
> > >
>
> >
>
> > > > > > ce
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > ssor.java:366)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.AbstractAutowireCapa
>
> > > > > > bl
>
> >
>
> > > > > > eB
>
> >
>
> > >
>
> >
>
> > > > > > ea
>
> >
>
> > >
>
> >
>
> > > > > > nF
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:12
>
> > > > > > 64
>
> >
>
> > > > > > )
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.AbstractAutowireCapa
>
> > > > > > bl
>
> >
>
> > > > > > eB
>
> >
>
> > >
>
> >
>
> > > > > > ea
>
> >
>
> > >
>
> >
>
> > > > > > nF
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:55
>
> > > > > > 3)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.AbstractAutowireCapa
>
> > > > > > bl
>
> >
>
> > > > > > eB
>
> >
>
> > >
>
> >
>
> > > > > > ea
>
> >
>
> > >
>
> >
>
> > > > > > nF
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.
>
> >
>
> > > > > > ge
>
> >
>
> > >
>
> >
>
> > > > > > tO
>
> >
>
> > >
>
> >
>
> > > > > > bj
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > ect(AbstractBeanFactory.java:306)
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > >
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > at
>
> >
>
> > >
>
> >
>
> > > > >
>
> >
>
> > >
>
> >
>
> > > > > > org.springframework.beans.factory.support.DefaultSingletonBean
>
> > > > > > Re
>
> >
>
> > > > > > gi
>
> >
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
I copied over users.xml, authorizers.xml and authorizations.xml to host-2, removed flow.xml.gz, and started NiFi there. Unfortunately, for whatever reason, the nodes still don’t talk to each other, even though both of them are connected to ZooKeeper on host-1. I still see two separate clusters, one on host-1 with all the dataflows, and the other, on host-2, without any of them. On the latter, the logs have no mention of host-1 whatsoever, neither server name, nor IP address. On host-1, nifi-app.log contains a few lines like the following:
2018-10-23 13:44:43,628 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Client attempting to establish new session at /<host-2 IP address>:50412
2018-10-23 13:44:43,629 INFO [SyncThread:0] o.a.zookeeper.server.ZooKeeperServer Established session 0x166a1d139590002 with negotiated timeout 4000 for client /<host-2 IP address>:50412
I apologize for bugging you with all this, converting our standalone NiFi instances into cluster nodes turned out to be much more challenging than we had anticipated…
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Tuesday, October 23, 2018 1:17 PM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
Probably easiest to copy the files over since you have other existing users/policies and you know the first node is working.
On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Tuesday, October 23, 2018 12:36 PM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> That means the user representing host-1 does not have permissions to proxy.
>
>
>
> You can look in authorizations.xml on nifi-1 for a policy like:
>
>
>
> <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> resource="/proxy" action="W">
>
> <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> </policy>
>
>
>
> That user identifier should point to a user in users.xml like:
>
>
>
> <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> Government, C=US"/>
>
>
>
> All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
>
>
> On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > Hi Bryan,
>
> >
>
> >
>
> >
>
> > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > · restarted one instance in the standalone mode
>
> >
>
> > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > · granted them the “proxy user requests” privileges
>
> >
>
> > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > · restarted the node on host-1
>
> >
>
> >
>
> >
>
> > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> >
>
> >
>
> > Insufficient Permissions
>
> >
>
> > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> >
>
> >
>
> > The nifi-user.log also contains
>
> >
>
> >
>
> >
>
> > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS,
>
> > O=U.S. Government, C=US
>
> >
>
> >
>
> >
>
> > From your experience, what the most likely causes for this exception?
>
> >
>
> >
>
> >
>
> > Thank you,
>
> >
>
> >
>
> >
>
> > Alexander
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>>
>
> > Sent: Monday, October 22, 2018 1:25 PM
>
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > Yes, to further clarify what I meant...
>
> >
>
> >
>
> >
>
> > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
> >
>
> >
>
> >
>
> > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
> >
>
> >
>
> >
>
> > - In standalone mode, use the UI to add users for the DN's of the
>
> > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2,
> > OU=NIFI)
>
> >
>
> > - In the UI, grant those users Write access to "Proxy"
>
> >
>
> > - Convert to a cluster and keep your same authorizers.xml,
> > users.xml,
>
> > and authorizations.xml when you setup your cluster, this way all
> > your
>
> > users and policies are already setup and the Initial Admin and Node
>
> > Identities are not needed
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > >
>
> >
>
> > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > Sent: Monday, October 22, 2018 12:48 PM
>
> >
>
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> >
>
> > >
>
> >
>
> > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> >
>
> > >
>
> >
>
> > > This blog post shows how to setup a secure 2 node cluster:
>
> >
>
> > >
>
> >
>
> > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-au
> > > th
>
> > > or
>
> >
>
> > > ization-and-multi-tenancy
>
> >
>
> > >
>
> >
>
> > > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> >
>
> > >
>
> >
>
> > > <authorizer>
>
> >
>
> > > <identifier>file-provider</identifier>
>
> >
>
> > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
>
> >
>
> > > <property name="Authorizations
>
> > > File">./conf/authorizations.xml</property>
>
> >
>
> > > <property name="Users File">./conf/users.xml</property>
>
> >
>
> > > <property name="Initial Admin Identity">CN=bbende,
>
> > > OU=ApacheNiFi</property>
>
> >
>
> > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > > <property name="Node Identity 1">CN=localhost,
>
> > > OU=NIFI</property>
>
> >
>
> > > </authorizer>
>
> >
>
> > >
>
> >
>
> > > You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> >
>
> > >
>
> >
>
> > > <userGroupProvider>
>
> >
>
> > > <identifier>file-user-group-provider</identifier>
>
> >
>
> > >
>
> > > <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
>
> >
>
> > > <property name="Users File">./conf/users.xml</property>
>
> >
>
> > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > > <property name="Initial User Identity 1">CN=bbende,
>
> > > OU=Apache NiFI</property>
>
> >
>
> > > <property name="Initial User Identity 2">CN=nifi-host-1,
>
> > > OU=NIFI</property>
>
> >
>
> > > <property name="Initial User Identity 2">CN=nifi-host-2,
>
> > > OU=NIFI</property>
>
> >
>
> > > </userGroupProvider>
>
> >
>
> > >
>
> >
>
> > > <accessPolicyProvider>
>
> >
>
> > > <identifier>file-access-policy-provider</identifier>
>
> >
>
> > >
>
> > > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</cla
> > > ss
>
> > > >
>
> >
>
> > > <property name="User Group
>
> >
>
> > > Provider">composite-configurable-user-group-provider</property>
>
> >
>
> > > <property name="Authorizations
>
> >
>
> > > File">./conf/authorizations.xml</property>
>
> >
>
> > > <property name="Initial Admin Identity">CN=bbende,
> > > OU=Apache
>
> > > NiFI</property>
>
> >
>
> > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > > <property name="Node Identity 1">CN=nifi-host-1,
>
> > > OU=NIFI</property>
>
> >
>
> > > <property name="Node Identity 1">CN=nifi-host-2,
>
> > > OU=NIFI</property>
>
> >
>
> > > </accessPolicyProvider>
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > Hi Bryan,
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > Are you saying that we have to run ZooKeeper on both nodes? BTW,
>
> > > > do
>
> >
>
> > > > we still need
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > nifi.login.identity.provider.configuration.file=./conf/login-ide
> > > > nt
>
> > > > it
>
> >
>
> > > > y-
>
> >
>
> > > > providers.xml
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > Thank you,
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > Alexander
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>>
>
> >
>
> > > > Sent: Monday, October 22, 2018 11:55 AM
>
> >
>
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > server.1=nifi-node1-hostname:2888:3888
>
> >
>
> > > >
>
> >
>
> > > > server.2=nifi-node2-hostname:2888:3888
>
> >
>
> > > >
>
> >
>
> > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > I wonder if anyone has run into the same problem when trying
> > > > > to
>
> >
>
> > > >
>
> >
>
> > > > > configure composite authentication/authorization (LDAP and
>
> > > > > local
>
> >
>
> > > >
>
> >
>
> > > > > file)? When we use the “stand-alone” authorizers.xml file with
>
> > > > > the
>
> >
>
> > > >
>
> >
>
> > > > > addition of two extra properties
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <property name="Node Identity 1">…
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <property name="Node Identity 2">…
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > and let ZooKeeper start on one on the nodes, we end up with
> > > > > two
>
> >
>
> > > >
>
> >
>
> > > > > one-node clusters, since apparently, the NiFi instances don’t
>
> > > > > talk
>
> >
>
> > > > > to
>
> >
>
> > > >
>
> >
>
> > > > > each other, but at least, they come alive…
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > From: Saip, Alexander (NIH/CC/BTRIS) [C]
>
> > > > > <al...@nih.gov>>
>
> >
>
> > > >
>
> >
>
> > > > > Sent: Friday, October 19, 2018 11:18 AM
>
> >
>
> > > >
>
> >
>
> > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> >
>
> > > >
>
> >
>
> > > > > Subject: RE: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > 2018-10-19 08:09:26,992 ERROR [main]
>
> > > > > o.s.web.context.ContextLoader
>
> >
>
> > > >
>
> >
>
> > > > > Context initialization failed
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> >
>
> > > >
>
> >
>
> > > > > Error creating bean with name
>
> >
>
> > > >
>
> >
>
> > > > > 'org.springframework.security.config.annotation.web.configuration.
>
> >
>
> > > > > We
>
> >
>
> > > > > bS
>
> >
>
> > > >
>
> >
>
> > > > > ecurityConfiguration': Unsatisfied dependency expressed
> > > > > through
>
> >
>
> > > > > method
>
> >
>
> > > >
>
> >
>
> > > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
>
> >
>
> > > > > exception
>
> >
>
> > > >
>
> >
>
> > > > > is org.springframework.beans.factory.BeanExpressionException:
>
> >
>
> > > >
>
> >
>
> > > > > Expression parsing failed; nested exception is
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> >
>
> > > >
>
> >
>
> > > > > Error creating bean with name
>
> >
>
> > > >
>
> >
>
> > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
>
> > > > > Unsatisfied
>
> >
>
> > > >
>
> >
>
> > > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> >
>
> > > >
>
> >
>
> > > > > parameter 0; nested exception is
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.BeanCreationException: Error
>
> >
>
> > > >
>
> >
>
> > > > > creating bean with name 'jwtAuthenticationProvider' defined in
>
> >
>
> > > > > class
>
> >
>
> > > >
>
> >
>
> > > > > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> >
>
> > > >
>
> >
>
> > > > > reference to bean 'authorizer' while setting constructor
>
> > > > > argument;
>
> >
>
> > > >
>
> >
>
> > > > > nested exception is
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.BeanCreationException: Error
>
> >
>
> > > >
>
> >
>
> > > > > creating bean with name 'authorizer': FactoryBean threw
>
> > > > > exception
>
> >
>
> > > > > on
>
> >
>
> > > >
>
> >
>
> > > > > object creation; nested exception is java.lang.NullPointerException:
>
> >
>
> > > >
>
> >
>
> > > > > Name is null
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.annotation.AutowiredAnnotati
> > > > > on
>
> > > > > Be
>
> >
>
> > > > > an
>
> >
>
> > > > > Po
>
> >
>
> > > >
>
> >
>
> > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationB
> > > > > ea
>
> > > > > nP
>
> >
>
> > > > > os
>
> >
>
> > > > > tP
>
> >
>
> > > >
>
> >
>
> > > > > rocessor.java:667)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.annotation.InjectionMetadata
> > > > > .i
>
> > > > > nj
>
> >
>
> > > > > ec
>
> >
>
> > > > > t(
>
> >
>
> > > >
>
> >
>
> > > > > InjectionMetadata.java:88)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.annotation.AutowiredAnnotati
> > > > > on
>
> > > > > Be
>
> >
>
> > > > > an
>
> >
>
> > > > > Po
>
> >
>
> > > >
>
> >
>
> > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanP
> > > > > os
>
> > > > > tP
>
> >
>
> > > > > ro
>
> >
>
> > > > > ce
>
> >
>
> > > >
>
> >
>
> > > > > ssor.java:366)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractAutowireCapa
> > > > > bl
>
> > > > > eB
>
> >
>
> > > > > ea
>
> >
>
> > > > > nF
>
> >
>
> > > >
>
> >
>
> > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:12
> > > > > 64
>
> > > > > )
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractAutowireCapa
> > > > > bl
>
> > > > > eB
>
> >
>
> > > > > ea
>
> >
>
> > > > > nF
>
> >
>
> > > >
>
> >
>
> > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:55
> > > > > 3)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractAutowireCapa
> > > > > bl
>
> > > > > eB
>
> >
>
> > > > > ea
>
> >
>
> > > > > nF
>
> >
>
> > > >
>
> >
>
> > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.
>
> > > > > ge
>
> >
>
> > > > > tO
>
> >
>
> > > > > bj
>
> >
>
> > > >
>
> >
>
> > > > > ect(AbstractBeanFactory.java:306)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.DefaultSingletonBean
> > > > > Re
>
> > > > > gi
>
> >
>
> > > > > st
>
> >
>
> > > > > ry
>
> >
>
> > > >
>
> >
>
> > > > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractBeanFactory.
> > > > > do
>
> > > > > Ge
>
> >
>
> > > > > tB
>
> >
>
> > > > > ea
>
> >
>
> > > >
>
> >
>
> > > > > n(AbstractBeanFactory.java:302)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractBeanFactory.
> > > > > ge
>
> > > > > tB
>
> >
>
> > > > > ea
>
> >
>
> > > > > n(
>
> >
>
> > > >
>
> >
>
> > > > > AbstractBeanFactory.java:197)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.DefaultListableBeanF
> > > > > ac
>
> > > > > to
>
> >
>
> > > > > ry
>
> >
>
> > > > > .p
>
> >
>
> > > >
>
> >
>
> > > > > reInstantiateSingletons(DefaultListableBeanFactory.java:761)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.context.support.AbstractApplicationContext
> > > > > .f
>
> > > > > in
>
> >
>
> > > > > is
>
> >
>
> > > > > hB
>
> >
>
> > > >
>
> >
>
> > > > > eanFactoryInitialization(AbstractApplicationContext.java:867)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.context.support.AbstractApplicationContext
> > > > > .r
>
> > > > > ef
>
> >
>
> > > > > re
>
> >
>
> > > > > sh
>
> >
>
> > > >
>
> >
>
> > > > > (AbstractApplicationContext.java:543)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.web.context.ContextLoader.configureAndRefr
> > > > > es
>
> > > > > hW
>
> >
>
> > > > > eb
>
> >
>
> > > > > Ap
>
> >
>
> > > >
>
> >
>
> > > > > plicationContext(ContextLoader.java:443)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.web.context.ContextLoader.initWebApplicati
> > > > > on
>
> > > > > Co
>
> >
>
> > > > > nt
>
> >
>
> > > > > ex
>
> >
>
> > > >
>
> >
>
> > > > > t(ContextLoader.java:325)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.web.context.ContextLoaderListener.contextI
> > > > > ni
>
> > > > > ti
>
> >
>
> > > > > al
>
> >
>
> > > > > iz
>
> >
>
> > > >
>
> >
>
> > > > > ed(ContextLoaderListener.java:107)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.ContextHandler.callContextIni
> > > > > ti
>
> > > > > al
>
> >
>
> > > > > iz
>
> >
>
> > > > > ed
>
> >
>
> > > >
>
> >
>
> > > > > (ContextHandler.java:876)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.servlet.ServletContextHandler.callContextIni
> > > > > ti
>
> > > > > al
>
> >
>
> > > > > iz
>
> >
>
> > > > > ed
>
> >
>
> > > >
>
> >
>
> > > > > (ServletContextHandler.java:532)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.ContextHandler.startContext(C
> > > > > on
>
> > > > > te
>
> >
>
> > > > > xt
>
> >
>
> > > > > Ha
>
> >
>
> > > >
>
> >
>
> > > > > ndler.java:839)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(S
> > > > > er
>
> > > > > vl
>
> >
>
> > > > > et
>
> >
>
> > > > > Co
>
> >
>
> > > >
>
> >
>
> > > > > ntextHandler.java:344)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:
>
> >
>
> > > >
>
> >
>
> > > > > 1480)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.
>
> >
>
> > > > > ja
>
> >
>
> > > > > va
>
> >
>
> > > >
>
> >
>
> > > > > :1442)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.ContextHandler.doStart(Contex
> > > > > tH
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > > > er
>
> >
>
> > > >
>
> >
>
> > > > > .java:799)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(Servle
> > > > > tC
>
> > > > > on
>
> >
>
> > > > > te
>
> >
>
> > > > > xt
>
> >
>
> > > >
>
> >
>
> > > > > Handler.java:261)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:
>
> >
>
> > > > > 54
>
> >
>
> > > > > 0)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstr
> > > > > ac
>
> > > > > tL
>
> >
>
> > > > > if
>
> >
>
> > > > > eC
>
> >
>
> > > >
>
> >
>
> > > > > ycle.java:68)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Cont
> > > > > ai
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > > > if
>
> >
>
> > > >
>
> >
>
> > > > > eCycle.java:131)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Co
> > > > > nt
>
> > > > > ai
>
> >
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > >
>
> >
>
> > > > > ifeCycle.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstr
> > > > > ac
>
> > > > > tH
>
> >
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > >
>
> >
>
> > > > > er.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstr
> > > > > ac
>
> > > > > tL
>
> >
>
> > > > > if
>
> >
>
> > > > > eC
>
> >
>
> > > >
>
> >
>
> > > > > ycle.java:68)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Cont
> > > > > ai
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > > > if
>
> >
>
> > > >
>
> >
>
> > > > > eCycle.java:131)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Co
> > > > > nt
>
> > > > > ai
>
> >
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > >
>
> >
>
> > > > > ifeCycle.java:105)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstr
> > > > > ac
>
> > > > > tH
>
> >
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > >
>
> >
>
> > > > > er.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.
>
> >
>
> > > >
>
> >
>
> > > > > java:290)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstr
> > > > > ac
>
> > > > > tL
>
> >
>
> > > > > if
>
> >
>
> > > > > eC
>
> >
>
> > > >
>
> >
>
> > > > > ycle.java:68)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Cont
> > > > > ai
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > > > if
>
> >
>
> > > >
>
> >
>
> > > > > eCycle.java:131)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Co
> > > > > nt
>
> > > > > ai
>
> >
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > >
>
> >
>
> > > > > ifeCycle.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstr
> > > > > ac
>
> > > > > tH
>
> >
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > >
>
> >
>
> > > > > er.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstr
> > > > > ac
>
> > > > > tL
>
> >
>
> > > > > if
>
> >
>
> > > > > eC
>
> >
>
> > > >
>
> >
>
> > > > > ycle.java:68)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Cont
> > > > > ai
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > > > if
>
> >
>
> > > >
>
> >
>
> > > > > eCycle.java:131)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> > > > > org.eclipse.jetty.server.Server.start(Server.java:452)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Co
> > > > > nt
>
> > > > > ai
>
> >
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > >
>
> >
>
> > > > > ifeCycle.java:105)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstr
> > > > > ac
>
> > > > > tH
>
> >
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > >
>
> >
>
> > > > > er.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > ><
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
Probably easiest to copy the files over since you have other existing
users/policies and you know the first node is working.
On Tue, Oct 23, 2018 at 1:12 PM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Tuesday, October 23, 2018 12:36 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> That means the user representing host-1 does not have permissions to proxy.
>
>
>
> You can look in authorizations.xml on nifi-1 for a policy like:
>
>
>
> <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
>
> resource="/proxy" action="W">
>
> <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
>
> </policy>
>
>
>
> That user identifier should point to a user in users.xml like:
>
>
>
> <user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
>
> identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
>
> Government, C=US"/>
>
>
>
> All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
>
>
>
> On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > Hi Bryan,
>
> >
>
> >
>
> >
>
> > Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> >
>
> > · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> >
>
> > · restarted one instance in the standalone mode
>
> >
>
> > · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> >
>
> > · granted them the “proxy user requests” privileges
>
> >
>
> > · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> >
>
> > · restarted the node on host-1
>
> >
>
> >
>
> >
>
> > On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
> >
>
> >
>
> >
>
> > Insufficient Permissions
>
> >
>
> > Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
> >
>
> >
>
> >
>
> > The nifi-user.log also contains
>
> >
>
> >
>
> >
>
> > 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
>
> > o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
>
> > Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS,
>
> > O=U.S. Government, C=US
>
> >
>
> >
>
> >
>
> > From your experience, what the most likely causes for this exception?
>
> >
>
> >
>
> >
>
> > Thank you,
>
> >
>
> >
>
> >
>
> > Alexander
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Monday, October 22, 2018 1:25 PM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > Yes, to further clarify what I meant...
>
> >
>
> >
>
> >
>
> > If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
> >
>
> >
>
> >
>
> > In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
> >
>
> >
>
> >
>
> > - In standalone mode, use the UI to add users for the DN's of the
>
> > server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2, OU=NIFI)
>
> >
>
> > - In the UI, grant those users Write access to "Proxy"
>
> >
>
> > - Convert to a cluster and keep your same authorizers.xml, users.xml,
>
> > and authorizations.xml when you setup your cluster, this way all your
>
> > users and policies are already setup and the Initial Admin and Node
>
> > Identities are not needed
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > Sent: Monday, October 22, 2018 12:48 PM
>
> >
>
> > > To: users@nifi.apache.org
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> >
>
> > >
>
> >
>
> > > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> >
>
> > >
>
> >
>
> > > This blog post shows how to setup a secure 2 node cluster:
>
> >
>
> > >
>
> >
>
> > > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-auth
>
> > > or
>
> >
>
> > > ization-and-multi-tenancy
>
> >
>
> > >
>
> >
>
> > > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> >
>
> > >
>
> >
>
> > > <authorizer>
>
> >
>
> > > <identifier>file-provider</identifier>
>
> >
>
> > > <class>org.apache.nifi.authorization.FileAuthorizer</class>
>
> >
>
> > > <property name="Authorizations
>
> > > File">./conf/authorizations.xml</property>
>
> >
>
> > > <property name="Users File">./conf/users.xml</property>
>
> >
>
> > > <property name="Initial Admin Identity">CN=bbende,
>
> > > OU=ApacheNiFi</property>
>
> >
>
> > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > > <property name="Node Identity 1">CN=localhost,
>
> > > OU=NIFI</property>
>
> >
>
> > > </authorizer>
>
> >
>
> > >
>
> >
>
> > > You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> >
>
> > >
>
> >
>
> > > <userGroupProvider>
>
> >
>
> > > <identifier>file-user-group-provider</identifier>
>
> >
>
> > >
>
> > > <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
>
> >
>
> > > <property name="Users File">./conf/users.xml</property>
>
> >
>
> > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > > <property name="Initial User Identity 1">CN=bbende,
>
> > > OU=Apache NiFI</property>
>
> >
>
> > > <property name="Initial User Identity 2">CN=nifi-host-1,
>
> > > OU=NIFI</property>
>
> >
>
> > > <property name="Initial User Identity 2">CN=nifi-host-2,
>
> > > OU=NIFI</property>
>
> >
>
> > > </userGroupProvider>
>
> >
>
> > >
>
> >
>
> > > <accessPolicyProvider>
>
> >
>
> > > <identifier>file-access-policy-provider</identifier>
>
> >
>
> > >
>
> > > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class
>
> > > >
>
> >
>
> > > <property name="User Group
>
> >
>
> > > Provider">composite-configurable-user-group-provider</property>
>
> >
>
> > > <property name="Authorizations
>
> >
>
> > > File">./conf/authorizations.xml</property>
>
> >
>
> > > <property name="Initial Admin Identity">CN=bbende, OU=Apache
>
> > > NiFI</property>
>
> >
>
> > > <property name="Legacy Authorized Users File"></property>
>
> >
>
> > > <property name="Node Identity 1">CN=nifi-host-1,
>
> > > OU=NIFI</property>
>
> >
>
> > > <property name="Node Identity 1">CN=nifi-host-2,
>
> > > OU=NIFI</property>
>
> >
>
> > > </accessPolicyProvider>
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >
>
> >
>
> > > > Hi Bryan,
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > Are you saying that we have to run ZooKeeper on both nodes? BTW,
>
> > > > do
>
> >
>
> > > > we still need
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > nifi.login.identity.provider.configuration.file=./conf/login-ident
>
> > > > it
>
> >
>
> > > > y-
>
> >
>
> > > > providers.xml
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > Thank you,
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > Alexander
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > -----Original Message-----
>
> >
>
> > > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > > Sent: Monday, October 22, 2018 11:55 AM
>
> >
>
> > > > To: users@nifi.apache.org
>
> >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > >
>
> >
>
> > > > server.1=nifi-node1-hostname:2888:3888
>
> >
>
> > > >
>
> >
>
> > > > server.2=nifi-node2-hostname:2888:3888
>
> >
>
> > > >
>
> >
>
> > > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > I wonder if anyone has run into the same problem when trying to
>
> >
>
> > > >
>
> >
>
> > > > > configure composite authentication/authorization (LDAP and
>
> > > > > local
>
> >
>
> > > >
>
> >
>
> > > > > file)? When we use the “stand-alone” authorizers.xml file with
>
> > > > > the
>
> >
>
> > > >
>
> >
>
> > > > > addition of two extra properties
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <property name="Node Identity 1">…
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > <property name="Node Identity 2">…
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > and let ZooKeeper start on one on the nodes, we end up with two
>
> >
>
> > > >
>
> >
>
> > > > > one-node clusters, since apparently, the NiFi instances don’t
>
> > > > > talk
>
> >
>
> > > > > to
>
> >
>
> > > >
>
> >
>
> > > > > each other, but at least, they come alive…
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > From: Saip, Alexander (NIH/CC/BTRIS) [C]
>
> > > > > <al...@nih.gov>
>
> >
>
> > > >
>
> >
>
> > > > > Sent: Friday, October 19, 2018 11:18 AM
>
> >
>
> > > >
>
> >
>
> > > > > To: users@nifi.apache.org
>
> >
>
> > > >
>
> >
>
> > > > > Subject: RE: NiFi fails on cluster nodes
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > 2018-10-19 08:09:26,992 ERROR [main]
>
> > > > > o.s.web.context.ContextLoader
>
> >
>
> > > >
>
> >
>
> > > > > Context initialization failed
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> >
>
> > > >
>
> >
>
> > > > > Error creating bean with name
>
> >
>
> > > >
>
> >
>
> > > > > 'org.springframework.security.config.annotation.web.configuration.
>
> >
>
> > > > > We
>
> >
>
> > > > > bS
>
> >
>
> > > >
>
> >
>
> > > > > ecurityConfiguration': Unsatisfied dependency expressed through
>
> >
>
> > > > > method
>
> >
>
> > > >
>
> >
>
> > > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
>
> >
>
> > > > > exception
>
> >
>
> > > >
>
> >
>
> > > > > is org.springframework.beans.factory.BeanExpressionException:
>
> >
>
> > > >
>
> >
>
> > > > > Expression parsing failed; nested exception is
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> >
>
> > > >
>
> >
>
> > > > > Error creating bean with name
>
> >
>
> > > >
>
> >
>
> > > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
>
> > > > > Unsatisfied
>
> >
>
> > > >
>
> >
>
> > > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> >
>
> > > >
>
> >
>
> > > > > parameter 0; nested exception is
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.BeanCreationException: Error
>
> >
>
> > > >
>
> >
>
> > > > > creating bean with name 'jwtAuthenticationProvider' defined in
>
> >
>
> > > > > class
>
> >
>
> > > >
>
> >
>
> > > > > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> >
>
> > > >
>
> >
>
> > > > > reference to bean 'authorizer' while setting constructor
>
> > > > > argument;
>
> >
>
> > > >
>
> >
>
> > > > > nested exception is
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.BeanCreationException: Error
>
> >
>
> > > >
>
> >
>
> > > > > creating bean with name 'authorizer': FactoryBean threw
>
> > > > > exception
>
> >
>
> > > > > on
>
> >
>
> > > >
>
> >
>
> > > > > object creation; nested exception is java.lang.NullPointerException:
>
> >
>
> > > >
>
> >
>
> > > > > Name is null
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.annotation.AutowiredAnnotation
>
> > > > > Be
>
> >
>
> > > > > an
>
> >
>
> > > > > Po
>
> >
>
> > > >
>
> >
>
> > > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBea
>
> > > > > nP
>
> >
>
> > > > > os
>
> >
>
> > > > > tP
>
> >
>
> > > >
>
> >
>
> > > > > rocessor.java:667)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.annotation.InjectionMetadata.i
>
> > > > > nj
>
> >
>
> > > > > ec
>
> >
>
> > > > > t(
>
> >
>
> > > >
>
> >
>
> > > > > InjectionMetadata.java:88)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.annotation.AutowiredAnnotation
>
> > > > > Be
>
> >
>
> > > > > an
>
> >
>
> > > > > Po
>
> >
>
> > > >
>
> >
>
> > > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPos
>
> > > > > tP
>
> >
>
> > > > > ro
>
> >
>
> > > > > ce
>
> >
>
> > > >
>
> >
>
> > > > > ssor.java:366)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractAutowireCapabl
>
> > > > > eB
>
> >
>
> > > > > ea
>
> >
>
> > > > > nF
>
> >
>
> > > >
>
> >
>
> > > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264
>
> > > > > )
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractAutowireCapabl
>
> > > > > eB
>
> >
>
> > > > > ea
>
> >
>
> > > > > nF
>
> >
>
> > > >
>
> >
>
> > > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractAutowireCapabl
>
> > > > > eB
>
> >
>
> > > > > ea
>
> >
>
> > > > > nF
>
> >
>
> > > >
>
> >
>
> > > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.
>
> > > > > ge
>
> >
>
> > > > > tO
>
> >
>
> > > > > bj
>
> >
>
> > > >
>
> >
>
> > > > > ect(AbstractBeanFactory.java:306)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.DefaultSingletonBeanRe
>
> > > > > gi
>
> >
>
> > > > > st
>
> >
>
> > > > > ry
>
> >
>
> > > >
>
> >
>
> > > > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractBeanFactory.do
>
> > > > > Ge
>
> >
>
> > > > > tB
>
> >
>
> > > > > ea
>
> >
>
> > > >
>
> >
>
> > > > > n(AbstractBeanFactory.java:302)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.AbstractBeanFactory.ge
>
> > > > > tB
>
> >
>
> > > > > ea
>
> >
>
> > > > > n(
>
> >
>
> > > >
>
> >
>
> > > > > AbstractBeanFactory.java:197)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.beans.factory.support.DefaultListableBeanFac
>
> > > > > to
>
> >
>
> > > > > ry
>
> >
>
> > > > > .p
>
> >
>
> > > >
>
> >
>
> > > > > reInstantiateSingletons(DefaultListableBeanFactory.java:761)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.context.support.AbstractApplicationContext.f
>
> > > > > in
>
> >
>
> > > > > is
>
> >
>
> > > > > hB
>
> >
>
> > > >
>
> >
>
> > > > > eanFactoryInitialization(AbstractApplicationContext.java:867)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.context.support.AbstractApplicationContext.r
>
> > > > > ef
>
> >
>
> > > > > re
>
> >
>
> > > > > sh
>
> >
>
> > > >
>
> >
>
> > > > > (AbstractApplicationContext.java:543)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.web.context.ContextLoader.configureAndRefres
>
> > > > > hW
>
> >
>
> > > > > eb
>
> >
>
> > > > > Ap
>
> >
>
> > > >
>
> >
>
> > > > > plicationContext(ContextLoader.java:443)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.web.context.ContextLoader.initWebApplication
>
> > > > > Co
>
> >
>
> > > > > nt
>
> >
>
> > > > > ex
>
> >
>
> > > >
>
> >
>
> > > > > t(ContextLoader.java:325)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.springframework.web.context.ContextLoaderListener.contextIni
>
> > > > > ti
>
> >
>
> > > > > al
>
> >
>
> > > > > iz
>
> >
>
> > > >
>
> >
>
> > > > > ed(ContextLoaderListener.java:107)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.ContextHandler.callContextIniti
>
> > > > > al
>
> >
>
> > > > > iz
>
> >
>
> > > > > ed
>
> >
>
> > > >
>
> >
>
> > > > > (ContextHandler.java:876)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.servlet.ServletContextHandler.callContextIniti
>
> > > > > al
>
> >
>
> > > > > iz
>
> >
>
> > > > > ed
>
> >
>
> > > >
>
> >
>
> > > > > (ServletContextHandler.java:532)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.ContextHandler.startContext(Con
>
> > > > > te
>
> >
>
> > > > > xt
>
> >
>
> > > > > Ha
>
> >
>
> > > >
>
> >
>
> > > > > ndler.java:839)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(Ser
>
> > > > > vl
>
> >
>
> > > > > et
>
> >
>
> > > > > Co
>
> >
>
> > > >
>
> >
>
> > > > > ntextHandler.java:344)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:
>
> >
>
> > > >
>
> >
>
> > > > > 1480)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.
>
> >
>
> > > > > ja
>
> >
>
> > > > > va
>
> >
>
> > > >
>
> >
>
> > > > > :1442)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextH
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > > > er
>
> >
>
> > > >
>
> >
>
> > > > > .java:799)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletC
>
> > > > > on
>
> >
>
> > > > > te
>
> >
>
> > > > > xt
>
> >
>
> > > >
>
> >
>
> > > > > Handler.java:261)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:
>
> >
>
> > > > > 54
>
> >
>
> > > > > 0)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstrac
>
> > > > > tL
>
> >
>
> > > > > if
>
> >
>
> > > > > eC
>
> >
>
> > > >
>
> >
>
> > > > > ycle.java:68)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Contai
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > > > if
>
> >
>
> > > >
>
> >
>
> > > > > eCycle.java:131)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Cont
>
> > > > > ai
>
> >
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > >
>
> >
>
> > > > > ifeCycle.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstrac
>
> > > > > tH
>
> >
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > >
>
> >
>
> > > > > er.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstrac
>
> > > > > tL
>
> >
>
> > > > > if
>
> >
>
> > > > > eC
>
> >
>
> > > >
>
> >
>
> > > > > ycle.java:68)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Contai
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > > > if
>
> >
>
> > > >
>
> >
>
> > > > > eCycle.java:131)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Cont
>
> > > > > ai
>
> >
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > >
>
> >
>
> > > > > ifeCycle.java:105)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstrac
>
> > > > > tH
>
> >
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > >
>
> >
>
> > > > > er.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.
>
> >
>
> > > >
>
> >
>
> > > > > java:290)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstrac
>
> > > > > tL
>
> >
>
> > > > > if
>
> >
>
> > > > > eC
>
> >
>
> > > >
>
> >
>
> > > > > ycle.java:68)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Contai
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > > > if
>
> >
>
> > > >
>
> >
>
> > > > > eCycle.java:131)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Cont
>
> > > > > ai
>
> >
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > >
>
> >
>
> > > > > ifeCycle.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstrac
>
> > > > > tH
>
> >
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > >
>
> >
>
> > > > > er.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstrac
>
> > > > > tL
>
> >
>
> > > > > if
>
> >
>
> > > > > eC
>
> >
>
> > > >
>
> >
>
> > > > > ycle.java:68)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Contai
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > > > if
>
> >
>
> > > >
>
> >
>
> > > > > eCycle.java:131)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> > > > > org.eclipse.jetty.server.Server.start(Server.java:452)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Cont
>
> > > > > ai
>
> >
>
> > > > > ne
>
> >
>
> > > > > rL
>
> >
>
> > > >
>
> >
>
> > > > > ifeCycle.java:105)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > >
>
> >
>
> > > > > at
>
> >
>
> > > >
>
> >
>
> > > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstrac
>
> > > > > tH
>
> >
>
> > > > > an
>
> >
>
> > > > > dl
>
> >
>
> > > >
>
> >
>
> > > > > er.java:113)
>
> >
>
> > > >
>
> >
>
> > > > >
>
> >
>
> > > ><
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Embarrassingly enough, there was a missing whitespace in the host DN in the users.xml file. Thank you so much for pointing me in the right direction! Now, in order to add another node, should I copy users.xml and authorizations.xml from the connected node to it, or remove them there instead?
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Tuesday, October 23, 2018 12:36 PM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
That means the user representing host-1 does not have permissions to proxy.
You can look in authorizations.xml on nifi-1 for a policy like:
<policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
resource="/proxy" action="W">
<user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
</policy>
That user identifier should point to a user in users.xml like:
<user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
Government, C=US"/>
All of the user identities are case sensitive and white space sensitive so make sure whatever is in users.xml is exactly what is shown in the logs.
On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> Hi Bryan,
>
>
>
> Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> · restarted one instance in the standalone mode
>
> · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> · granted them the “proxy user requests” privileges
>
> · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> · restarted the node on host-1
>
>
>
> On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
>
>
> Insufficient Permissions
>
> Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
>
>
> The nifi-user.log also contains
>
>
>
> 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224]
> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:
> Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS,
> O=U.S. Government, C=US
>
>
>
> From your experience, what the most likely causes for this exception?
>
>
>
> Thank you,
>
>
>
> Alexander
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Monday, October 22, 2018 1:25 PM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Yes, to further clarify what I meant...
>
>
>
> If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
>
>
> In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
>
>
> - In standalone mode, use the UI to add users for the DN's of the
> server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2, OU=NIFI)
>
> - In the UI, grant those users Write access to "Proxy"
>
> - Convert to a cluster and keep your same authorizers.xml, users.xml,
> and authorizations.xml when you setup your cluster, this way all your
> users and policies are already setup and the Initial Admin and Node
> Identities are not needed
>
>
>
>
>
> On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>>
>
> > Sent: Monday, October 22, 2018 12:48 PM
>
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> >
>
> > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> >
>
> > This blog post shows how to setup a secure 2 node cluster:
>
> >
>
> > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-auth
> > or
>
> > ization-and-multi-tenancy
>
> >
>
> > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> >
>
> > <authorizer>
>
> > <identifier>file-provider</identifier>
>
> > <class>org.apache.nifi.authorization.FileAuthorizer</class>
>
> > <property name="Authorizations
> > File">./conf/authorizations.xml</property>
>
> > <property name="Users File">./conf/users.xml</property>
>
> > <property name="Initial Admin Identity">CN=bbende,
> > OU=ApacheNiFi</property>
>
> > <property name="Legacy Authorized Users File"></property>
>
> > <property name="Node Identity 1">CN=localhost,
> > OU=NIFI</property>
>
> > </authorizer>
>
> >
>
> > You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> >
>
> > <userGroupProvider>
>
> > <identifier>file-user-group-provider</identifier>
>
> >
> > <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
>
> > <property name="Users File">./conf/users.xml</property>
>
> > <property name="Legacy Authorized Users File"></property>
>
> > <property name="Initial User Identity 1">CN=bbende,
> > OU=Apache NiFI</property>
>
> > <property name="Initial User Identity 2">CN=nifi-host-1,
> > OU=NIFI</property>
>
> > <property name="Initial User Identity 2">CN=nifi-host-2,
> > OU=NIFI</property>
>
> > </userGroupProvider>
>
> >
>
> > <accessPolicyProvider>
>
> > <identifier>file-access-policy-provider</identifier>
>
> >
> > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class
> > >
>
> > <property name="User Group
>
> > Provider">composite-configurable-user-group-provider</property>
>
> > <property name="Authorizations
>
> > File">./conf/authorizations.xml</property>
>
> > <property name="Initial Admin Identity">CN=bbende, OU=Apache
> > NiFI</property>
>
> > <property name="Legacy Authorized Users File"></property>
>
> > <property name="Node Identity 1">CN=nifi-host-1,
> > OU=NIFI</property>
>
> > <property name="Node Identity 1">CN=nifi-host-2,
> > OU=NIFI</property>
>
> > </accessPolicyProvider>
>
> >
>
> >
>
> > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > Hi Bryan,
>
> > >
>
> > >
>
> > >
>
> > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
> > >
>
> > >
>
> > >
>
> > > Are you saying that we have to run ZooKeeper on both nodes? BTW,
> > > do
>
> > > we still need
>
> > >
>
> > >
>
> > >
>
> > > nifi.login.identity.provider.configuration.file=./conf/login-ident
> > > it
>
> > > y-
>
> > > providers.xml
>
> > >
>
> > >
>
> > >
>
> > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
> > >
>
> > >
>
> > >
>
> > > Thank you,
>
> > >
>
> > >
>
> > >
>
> > > Alexander
>
> > >
>
> > >
>
> > >
>
> > > -----Original Message-----
>
> > > From: Bryan Bende <bb...@gmail.com>>
>
> > > Sent: Monday, October 22, 2018 11:55 AM
>
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > >
>
> > >
>
> > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
> > >
>
> > >
>
> > >
>
> > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
> > >
>
> > >
>
> > >
>
> > > server.1=nifi-node1-hostname:2888:3888
>
> > >
>
> > > server.2=nifi-node2-hostname:2888:3888
>
> > >
>
> > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > I wonder if anyone has run into the same problem when trying to
>
> > >
>
> > > > configure composite authentication/authorization (LDAP and
> > > > local
>
> > >
>
> > > > file)? When we use the “stand-alone” authorizers.xml file with
> > > > the
>
> > >
>
> > > > addition of two extra properties
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > <property name="Node Identity 1">…
>
> > >
>
> > > >
>
> > >
>
> > > > <property name="Node Identity 2">…
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > and let ZooKeeper start on one on the nodes, we end up with two
>
> > >
>
> > > > one-node clusters, since apparently, the NiFi instances don’t
> > > > talk
>
> > > > to
>
> > >
>
> > > > each other, but at least, they come alive…
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > From: Saip, Alexander (NIH/CC/BTRIS) [C]
> > > > <al...@nih.gov>>
>
> > >
>
> > > > Sent: Friday, October 19, 2018 11:18 AM
>
> > >
>
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >
>
> > > > Subject: RE: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > 2018-10-19 08:09:26,992 ERROR [main]
> > > > o.s.web.context.ContextLoader
>
> > >
>
> > > > Context initialization failed
>
> > >
>
> > > >
>
> > >
>
> > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > >
>
> > > > Error creating bean with name
>
> > >
>
> > > > 'org.springframework.security.config.annotation.web.configuration.
>
> > > > We
>
> > > > bS
>
> > >
>
> > > > ecurityConfiguration': Unsatisfied dependency expressed through
>
> > > > method
>
> > >
>
> > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
>
> > > > exception
>
> > >
>
> > > > is org.springframework.beans.factory.BeanExpressionException:
>
> > >
>
> > > > Expression parsing failed; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > >
>
> > > > Error creating bean with name
>
> > >
>
> > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
> > > > Unsatisfied
>
> > >
>
> > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > >
>
> > > > parameter 0; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'jwtAuthenticationProvider' defined in
>
> > > > class
>
> > >
>
> > > > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > >
>
> > > > reference to bean 'authorizer' while setting constructor
> > > > argument;
>
> > >
>
> > > > nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'authorizer': FactoryBean threw
> > > > exception
>
> > > > on
>
> > >
>
> > > > object creation; nested exception is java.lang.NullPointerException:
>
> > >
>
> > > > Name is null
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotation
> > > > Be
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBea
> > > > nP
>
> > > > os
>
> > > > tP
>
> > >
>
> > > > rocessor.java:667)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.InjectionMetadata.i
> > > > nj
>
> > > > ec
>
> > > > t(
>
> > >
>
> > > > InjectionMetadata.java:88)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotation
> > > > Be
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPos
> > > > tP
>
> > > > ro
>
> > > > ce
>
> > >
>
> > > > ssor.java:366)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapabl
> > > > eB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264
> > > > )
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapabl
> > > > eB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapabl
> > > > eB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.
> > > > ge
>
> > > > tO
>
> > > > bj
>
> > >
>
> > > > ect(AbstractBeanFactory.java:306)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultSingletonBeanRe
> > > > gi
>
> > > > st
>
> > > > ry
>
> > >
>
> > > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.do
> > > > Ge
>
> > > > tB
>
> > > > ea
>
> > >
>
> > > > n(AbstractBeanFactory.java:302)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.ge
> > > > tB
>
> > > > ea
>
> > > > n(
>
> > >
>
> > > > AbstractBeanFactory.java:197)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFac
> > > > to
>
> > > > ry
>
> > > > .p
>
> > >
>
> > > > reInstantiateSingletons(DefaultListableBeanFactory.java:761)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.context.support.AbstractApplicationContext.f
> > > > in
>
> > > > is
>
> > > > hB
>
> > >
>
> > > > eanFactoryInitialization(AbstractApplicationContext.java:867)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.context.support.AbstractApplicationContext.r
> > > > ef
>
> > > > re
>
> > > > sh
>
> > >
>
> > > > (AbstractApplicationContext.java:543)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.web.context.ContextLoader.configureAndRefres
> > > > hW
>
> > > > eb
>
> > > > Ap
>
> > >
>
> > > > plicationContext(ContextLoader.java:443)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.web.context.ContextLoader.initWebApplication
> > > > Co
>
> > > > nt
>
> > > > ex
>
> > >
>
> > > > t(ContextLoader.java:325)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.web.context.ContextLoaderListener.contextIni
> > > > ti
>
> > > > al
>
> > > > iz
>
> > >
>
> > > > ed(ContextLoaderListener.java:107)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.ContextHandler.callContextIniti
> > > > al
>
> > > > iz
>
> > > > ed
>
> > >
>
> > > > (ContextHandler.java:876)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.servlet.ServletContextHandler.callContextIniti
> > > > al
>
> > > > iz
>
> > > > ed
>
> > >
>
> > > > (ServletContextHandler.java:532)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.ContextHandler.startContext(Con
> > > > te
>
> > > > xt
>
> > > > Ha
>
> > >
>
> > > > ndler.java:839)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(Ser
> > > > vl
>
> > > > et
>
> > > > Co
>
> > >
>
> > > > ntextHandler.java:344)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:
>
> > >
>
> > > > 1480)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.
>
> > > > ja
>
> > > > va
>
> > >
>
> > > > :1442)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextH
> > > > an
>
> > > > dl
>
> > > > er
>
> > >
>
> > > > .java:799)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletC
> > > > on
>
> > > > te
>
> > > > xt
>
> > >
>
> > > > Handler.java:261)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:
>
> > > > 54
>
> > > > 0)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstrac
> > > > tL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Contai
> > > > ne
>
> > > > rL
>
> > > > if
>
> > >
>
> > > > eCycle.java:131)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Cont
> > > > ai
>
> > > > ne
>
> > > > rL
>
> > >
>
> > > > ifeCycle.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstrac
> > > > tH
>
> > > > an
>
> > > > dl
>
> > >
>
> > > > er.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstrac
> > > > tL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Contai
> > > > ne
>
> > > > rL
>
> > > > if
>
> > >
>
> > > > eCycle.java:131)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Cont
> > > > ai
>
> > > > ne
>
> > > > rL
>
> > >
>
> > > > ifeCycle.java:105)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstrac
> > > > tH
>
> > > > an
>
> > > > dl
>
> > >
>
> > > > er.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.
>
> > >
>
> > > > java:290)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstrac
> > > > tL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Contai
> > > > ne
>
> > > > rL
>
> > > > if
>
> > >
>
> > > > eCycle.java:131)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Cont
> > > > ai
>
> > > > ne
>
> > > > rL
>
> > >
>
> > > > ifeCycle.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstrac
> > > > tH
>
> > > > an
>
> > > > dl
>
> > >
>
> > > > er.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstrac
> > > > tL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Contai
> > > > ne
>
> > > > rL
>
> > > > if
>
> > >
>
> > > > eCycle.java:131)
>
> > >
>
> > > >
>
> > >
>
> > > > at
> > > > org.eclipse.jetty.server.Server.start(Server.java:452)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Cont
> > > > ai
>
> > > > ne
>
> > > > rL
>
> > >
>
> > > > ifeCycle.java:105)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abstrac
> > > > tH
>
> > > > an
>
> > > > dl
>
> > >
>
> > > > er.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > > > org.eclipse.jetty.server.Server.doStart(Server.java:419)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abstrac
> > > > tL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:83
> > > > 8)
>
> > >
>
> > > >
>
> > >
>
> > > > at org.apache.nifi.NiFi.<init>(NiFi.java:157)
>
> > >
>
> > > >
>
> > >
>
> > > > at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> > >
>
> > > >
>
> > >
>
> > > > at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> > >
>
> > > >
>
> > >
>
> > > > Caused by: org.springframework.beans.factory.BeanExpressionException:
>
> > >
>
> > > > Expression parsing failed; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > >
>
> > > > Error creating bean with name
>
> > >
>
> > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
> > > > Unsatisfied
>
> > >
>
> > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > >
>
> > > > parameter 0; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'jwtAuthenticationProvider' defined in
>
> > > > class
>
> > >
>
> > > > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > >
>
> > > > reference to bean 'authorizer' while setting constructor
> > > > argument;
>
> > >
>
> > > > nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'authorizer': FactoryBean threw
> > > > exception
>
> > > > on
>
> > >
>
> > > > object creation; nested exception is java.lang.NullPointerException:
>
> > >
>
> > > > Name is null
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.context.expression.StandardBeanExpressionResolver.
>
> > >
>
> > > > evaluate(StandardBeanExpressionResolver.java:164)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.ev
> > > > al
>
> > > > ua
>
> > > > te
>
> > >
>
> > > > BeanDefinitionString(AbstractBeanFactory.java:1448)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFac
> > > > to
>
> > > > ry
>
> > > > .d
>
> > >
>
> > > > oResolveDependency(DefaultListableBeanFactory.java:1088)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFac
> > > > to
>
> > > > ry
>
> > > > .r
>
> > >
>
> > > > esolveDependency(DefaultListableBeanFactory.java:1066)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotation
> > > > Be
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBea
> > > > nP
>
> > > > os
>
> > > > tP
>
> > >
>
> > > > rocessor.java:659)
>
> > >
>
> > > >
>
> > >
>
> > > > ... 48 common frames omitted
>
> > >
>
> > > >
>
> > >
>
> > > > Caused by:
>
> > >
>
> > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > >
>
> > > > Error creating bean with name
>
> > >
>
> > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration':
> > > > Unsatisfied
>
> > >
>
> > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > >
>
> > > > parameter 0; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'jwtAuthenticationProvider' defined in
>
> > > > class
>
> > >
>
> > > > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > >
>
> > > > reference to bean 'authorizer' while setting constructor
> > > > argument;
>
> > >
>
> > > > nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'authorizer': FactoryBean threw
> > > > exception
>
> > > > on
>
> > >
>
> > > > object creation; nested exception is java.lang.NullPointerException:
>
> > >
>
> > > > Name is null
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotation
> > > > Be
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBea
> > > > nP
>
> > > > os
>
> > > > tP
>
> > >
>
> > > > rocessor.java:667)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.InjectionMetadata.i
> > > > nj
>
> > > > ec
>
> > > > t(
>
> > >
>
> > > > InjectionMetadata.java:88)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotation
> > > > Be
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPos
> > > > tP
>
> > > > ro
>
> > > > ce
>
> > >
>
> > > > ssor.java:366)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapabl
> > > > eB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264
> > > > )
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapabl
> > > > eB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapabl
> > > > eB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.
> > > > ge
>
> > > > tO
>
> > > > bj
>
> > >
>
> > > > ect(AbstractBeanFactory.java:306)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultSingletonBeanRe
> > > > gi
>
> > > > st
>
> > > > ry
>
> > >
>
> > > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.do
> > > > Ge
>
> > > > tB
>
> > > > ea
>
> > >
>
> > > > n(AbstractBeanFactory.java:302)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.ge
> > > > tB
>
> > > > ea
>
> > > > n(
>
> > >
>
> > > > AbstractBeanFactory.java:202)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFac
> > > > to
>
> > > > ry
>
> > > > .g
>
> > >
>
> > > > etBeansOfType(DefaultListableBeanFactory.java:519)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFac
> > > > to
>
> > > > ry
>
> > > > .g
>
> > >
>
> > > > etBeansOfType(DefaultListableBeanFactory.java:508)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.security.config.annotation.web.configuration
> > > > .A
>
> > > > ut
>
> > > > ow
>
> > >
>
> > > > iredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurer
> > > > s(
>
> > > > Au
>
> > > > to
>
> > >
>
> > > > wiredWebSecurityConfigurersIgnoreParents.java:53)
>
> > >
>
> > > >
>
> > >
>
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>
> > > > Method)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor
> > > > Im
>
> > > > pl
>
> > > > .j
>
> > >
>
> > > > ava:62)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
> > > > Ac
>
> > > > ce
>
> > > > ss
>
> > >
>
> > > > orImpl.java:43)
>
> > >
>
> > > >
>
> > >
>
> > > > at java.lang.reflect.Method.invoke(Method.java:498)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.support.ReflectiveMethodExec
> > > > ut
>
> > > > or
>
> > > > .e
>
> > >
>
> > > > xecute(ReflectiveMethodExecutor.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.MethodReference.getValue
> > > > In
>
> > > > te
>
> > > > rn
>
> > >
>
> > > > al(MethodReference.java:129)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.MethodReference.access$0
> > > > 00
>
> > > > (M
>
> > > > et
>
> > >
>
> > > > hodReference.java:49)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.MethodReference$MethodVa
> > > > lu
>
> > > > eR
>
> > > > ef
>
> > >
>
> > > > .getValue(MethodReference.java:347)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.CompoundExpression.getVa
> > > > lu
>
> > > > eI
>
> > > > nt
>
> > >
>
> > > > ernal(CompoundExpression.java:88)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.SpelNodeImpl.getValue(Sp
> > > > el
>
> > > > No
>
> > > > de
>
> > >
>
> > > > Impl.java:120)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.standard.SpelExpression.getV
> > > > al
>
> > > > ue
>
> > > > (S
>
> > >
>
> > > > pelExpression.java:262)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.context.expression.StandardBeanExpressionResolver.
>
> > >
>
> > > > evaluate(StandardBeanExpressionResolver.java:161)
>
> > >
>
> > > >
>
> > >
>
> > > > ... 52 common frames omitted
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > I tried to Google for possible clues, but so far, there hasn’t
>
> > > > been
>
> > >
>
> > > > any luck…
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > -----Original Message-----
>
> > >
>
> > > > From: Bryan Bende <bb...@gmail.com>>
>
> > >
>
> > > > Sent: Monday, October 15, 2018 10:27 AM
>
> > >
>
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
>
> > >
>
> > > >
>
> > >
>
> > > > On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > -----Original Message-----
>
> > >
>
> > > >
>
> > >
>
> > > > > From: Bryan Bende <bb...@gmail.com>>
>
> > >
>
> > > >
>
> > >
>
> > > > > Sent: Monday, October 15, 2018 9:43 AM
>
> > >
>
> > > >
>
> > >
>
> > > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >
>
> > > >
>
> > >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com>> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Mike
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >> Hi Mike and Bryan,
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol
>
> > >
>
> > > > > >> Request-1]
>
> > >
>
> > > >
>
> > >
>
> > > > > >> o.a.nifi.security.util.CertificateUtils The incoming
> > > > > >> request
>
> > > > > >> did
>
> > >
>
> > > >
>
> > >
>
> > > > > >> not contain client certificates and thus the DN cannot be extracted.
>
> > >
>
> > > >
>
> > >
>
> > > > > >> Check that the other endp
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
That means the user representing host-1 does not have permissions to proxy.
You can look in authorizations.xml on nifi-1 for a policy like:
<policy identifier="287edf48-da72-359b-8f61-da5d4c45a270"
resource="/proxy" action="W">
<user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"/>
</policy>
That user identifier should point to a user in users.xml like:
<user identifier="c22273fa-7ed3-38a9-8994-3ed5fea5d234"
identity="CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S.
Government, C=US"/>
All of the user identities are case sensitive and white space
sensitive so make sure whatever is in users.xml is exactly what is
shown in the logs.
On Tue, Oct 23, 2018 at 12:28 PM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> Hi Bryan,
>
>
>
> Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
>
> · restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
>
> · restarted one instance in the standalone mode
>
> · added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
>
> · granted them the “proxy user requests” privileges
>
> · edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
>
> · restarted the node on host-1
>
>
>
> On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
>
>
>
> Insufficient Permissions
>
> Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
>
>
>
> The nifi-user.log also contains
>
>
>
> 2018-10-23 12:17:01,916 WARN [NiFi Web Server-224] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US
>
>
>
> From your experience, what the most likely causes for this exception?
>
>
>
> Thank you,
>
>
>
> Alexander
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Monday, October 22, 2018 1:25 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Yes, to further clarify what I meant...
>
>
>
> If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
>
>
>
> In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
>
>
>
> - In standalone mode, use the UI to add users for the DN's of the server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2, OU=NIFI)
>
> - In the UI, grant those users Write access to "Proxy"
>
> - Convert to a cluster and keep your same authorizers.xml, users.xml, and authorizations.xml when you setup your cluster, this way all your users and policies are already setup and the Initial Admin and Node Identities are not needed
>
>
>
>
>
> On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Monday, October 22, 2018 12:48 PM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> >
>
> > You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> >
>
> > This blog post shows how to setup a secure 2 node cluster:
>
> >
>
> > https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-author
>
> > ization-and-multi-tenancy
>
> >
>
> > The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> >
>
> > <authorizer>
>
> > <identifier>file-provider</identifier>
>
> > <class>org.apache.nifi.authorization.FileAuthorizer</class>
>
> > <property name="Authorizations File">./conf/authorizations.xml</property>
>
> > <property name="Users File">./conf/users.xml</property>
>
> > <property name="Initial Admin Identity">CN=bbende, OU=ApacheNiFi</property>
>
> > <property name="Legacy Authorized Users File"></property>
>
> > <property name="Node Identity 1">CN=localhost, OU=NIFI</property>
>
> > </authorizer>
>
> >
>
> > You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> >
>
> > <userGroupProvider>
>
> > <identifier>file-user-group-provider</identifier>
>
> > <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
>
> > <property name="Users File">./conf/users.xml</property>
>
> > <property name="Legacy Authorized Users File"></property>
>
> > <property name="Initial User Identity 1">CN=bbende, OU=Apache NiFI</property>
>
> > <property name="Initial User Identity 2">CN=nifi-host-1, OU=NIFI</property>
>
> > <property name="Initial User Identity 2">CN=nifi-host-2, OU=NIFI</property>
>
> > </userGroupProvider>
>
> >
>
> > <accessPolicyProvider>
>
> > <identifier>file-access-policy-provider</identifier>
>
> > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>
> > <property name="User Group
>
> > Provider">composite-configurable-user-group-provider</property>
>
> > <property name="Authorizations
>
> > File">./conf/authorizations.xml</property>
>
> > <property name="Initial Admin Identity">CN=bbende, OU=Apache NiFI</property>
>
> > <property name="Legacy Authorized Users File"></property>
>
> > <property name="Node Identity 1">CN=nifi-host-1, OU=NIFI</property>
>
> > <property name="Node Identity 1">CN=nifi-host-2, OU=NIFI</property>
>
> > </accessPolicyProvider>
>
> >
>
> >
>
> > Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > Hi Bryan,
>
> > >
>
> > >
>
> > >
>
> > > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
> > >
>
> > >
>
> > >
>
> > > Are you saying that we have to run ZooKeeper on both nodes? BTW, do
>
> > > we still need
>
> > >
>
> > >
>
> > >
>
> > > nifi.login.identity.provider.configuration.file=./conf/login-identit
>
> > > y-
>
> > > providers.xml
>
> > >
>
> > >
>
> > >
>
> > > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
> > >
>
> > >
>
> > >
>
> > > Thank you,
>
> > >
>
> > >
>
> > >
>
> > > Alexander
>
> > >
>
> > >
>
> > >
>
> > > -----Original Message-----
>
> > > From: Bryan Bende <bb...@gmail.com>
>
> > > Sent: Monday, October 22, 2018 11:55 AM
>
> > > To: users@nifi.apache.org
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > >
>
> > >
>
> > > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
> > >
>
> > >
>
> > >
>
> > > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
> > >
>
> > >
>
> > >
>
> > > server.1=nifi-node1-hostname:2888:3888
>
> > >
>
> > > server.2=nifi-node2-hostname:2888:3888
>
> > >
>
> > > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > I wonder if anyone has run into the same problem when trying to
>
> > >
>
> > > > configure composite authentication/authorization (LDAP and local
>
> > >
>
> > > > file)? When we use the “stand-alone” authorizers.xml file with the
>
> > >
>
> > > > addition of two extra properties
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > <property name="Node Identity 1">…
>
> > >
>
> > > >
>
> > >
>
> > > > <property name="Node Identity 2">…
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > and let ZooKeeper start on one on the nodes, we end up with two
>
> > >
>
> > > > one-node clusters, since apparently, the NiFi instances don’t talk
>
> > > > to
>
> > >
>
> > > > each other, but at least, they come alive…
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
>
> > >
>
> > > > Sent: Friday, October 19, 2018 11:18 AM
>
> > >
>
> > > > To: users@nifi.apache.org
>
> > >
>
> > > > Subject: RE: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > 2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader
>
> > >
>
> > > > Context initialization failed
>
> > >
>
> > > >
>
> > >
>
> > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > >
>
> > > > Error creating bean with name
>
> > >
>
> > > > 'org.springframework.security.config.annotation.web.configuration.
>
> > > > We
>
> > > > bS
>
> > >
>
> > > > ecurityConfiguration': Unsatisfied dependency expressed through
>
> > > > method
>
> > >
>
> > > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
>
> > > > exception
>
> > >
>
> > > > is org.springframework.beans.factory.BeanExpressionException:
>
> > >
>
> > > > Expression parsing failed; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > >
>
> > > > Error creating bean with name
>
> > >
>
> > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>
> > >
>
> > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > >
>
> > > > parameter 0; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'jwtAuthenticationProvider' defined in
>
> > > > class
>
> > >
>
> > > > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > >
>
> > > > reference to bean 'authorizer' while setting constructor argument;
>
> > >
>
> > > > nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'authorizer': FactoryBean threw exception
>
> > > > on
>
> > >
>
> > > > object creation; nested exception is java.lang.NullPointerException:
>
> > >
>
> > > > Name is null
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanP
>
> > > > os
>
> > > > tP
>
> > >
>
> > > > rocessor.java:667)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.InjectionMetadata.inj
>
> > > > ec
>
> > > > t(
>
> > >
>
> > > > InjectionMetadata.java:88)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostP
>
> > > > ro
>
> > > > ce
>
> > >
>
> > > > ssor.java:366)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapableB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapableB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapableB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.ge
>
> > > > tO
>
> > > > bj
>
> > >
>
> > > > ect(AbstractBeanFactory.java:306)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultSingletonBeanRegi
>
> > > > st
>
> > > > ry
>
> > >
>
> > > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.doGe
>
> > > > tB
>
> > > > ea
>
> > >
>
> > > > n(AbstractBeanFactory.java:302)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.getB
>
> > > > ea
>
> > > > n(
>
> > >
>
> > > > AbstractBeanFactory.java:197)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFacto
>
> > > > ry
>
> > > > .p
>
> > >
>
> > > > reInstantiateSingletons(DefaultListableBeanFactory.java:761)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.context.support.AbstractApplicationContext.fin
>
> > > > is
>
> > > > hB
>
> > >
>
> > > > eanFactoryInitialization(AbstractApplicationContext.java:867)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.context.support.AbstractApplicationContext.ref
>
> > > > re
>
> > > > sh
>
> > >
>
> > > > (AbstractApplicationContext.java:543)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.web.context.ContextLoader.configureAndRefreshW
>
> > > > eb
>
> > > > Ap
>
> > >
>
> > > > plicationContext(ContextLoader.java:443)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.web.context.ContextLoader.initWebApplicationCo
>
> > > > nt
>
> > > > ex
>
> > >
>
> > > > t(ContextLoader.java:325)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.web.context.ContextLoaderListener.contextIniti
>
> > > > al
>
> > > > iz
>
> > >
>
> > > > ed(ContextLoaderListener.java:107)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.ContextHandler.callContextInitial
>
> > > > iz
>
> > > > ed
>
> > >
>
> > > > (ContextHandler.java:876)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitial
>
> > > > iz
>
> > > > ed
>
> > >
>
> > > > (ServletContextHandler.java:532)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.ContextHandler.startContext(Conte
>
> > > > xt
>
> > > > Ha
>
> > >
>
> > > > ndler.java:839)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(Servl
>
> > > > et
>
> > > > Co
>
> > >
>
> > > > ntextHandler.java:344)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:
>
> > >
>
> > > > 1480)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.
>
> > > > ja
>
> > > > va
>
> > >
>
> > > > :1442)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHan
>
> > > > dl
>
> > > > er
>
> > >
>
> > > > .java:799)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletCon
>
> > > > te
>
> > > > xt
>
> > >
>
> > > > Handler.java:261)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:
>
> > > > 54
>
> > > > 0)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Containe
>
> > > > rL
>
> > > > if
>
> > >
>
> > > > eCycle.java:131)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Contai
>
> > > > ne
>
> > > > rL
>
> > >
>
> > > > ifeCycle.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractH
>
> > > > an
>
> > > > dl
>
> > >
>
> > > > er.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Containe
>
> > > > rL
>
> > > > if
>
> > >
>
> > > > eCycle.java:131)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Contai
>
> > > > ne
>
> > > > rL
>
> > >
>
> > > > ifeCycle.java:105)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractH
>
> > > > an
>
> > > > dl
>
> > >
>
> > > > er.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.
>
> > >
>
> > > > java:290)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Containe
>
> > > > rL
>
> > > > if
>
> > >
>
> > > > eCycle.java:131)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Contai
>
> > > > ne
>
> > > > rL
>
> > >
>
> > > > ifeCycle.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractH
>
> > > > an
>
> > > > dl
>
> > >
>
> > > > er.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Containe
>
> > > > rL
>
> > > > if
>
> > >
>
> > > > eCycle.java:131)
>
> > >
>
> > > >
>
> > >
>
> > > > at org.eclipse.jetty.server.Server.start(Server.java:452)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Contai
>
> > > > ne
>
> > > > rL
>
> > >
>
> > > > ifeCycle.java:105)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractH
>
> > > > an
>
> > > > dl
>
> > >
>
> > > > er.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > > > org.eclipse.jetty.server.Server.doStart(Server.java:419)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
>
> > > > if
>
> > > > eC
>
> > >
>
> > > > ycle.java:68)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838)
>
> > >
>
> > > >
>
> > >
>
> > > > at org.apache.nifi.NiFi.<init>(NiFi.java:157)
>
> > >
>
> > > >
>
> > >
>
> > > > at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> > >
>
> > > >
>
> > >
>
> > > > at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> > >
>
> > > >
>
> > >
>
> > > > Caused by: org.springframework.beans.factory.BeanExpressionException:
>
> > >
>
> > > > Expression parsing failed; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > >
>
> > > > Error creating bean with name
>
> > >
>
> > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>
> > >
>
> > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > >
>
> > > > parameter 0; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'jwtAuthenticationProvider' defined in
>
> > > > class
>
> > >
>
> > > > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > >
>
> > > > reference to bean 'authorizer' while setting constructor argument;
>
> > >
>
> > > > nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'authorizer': FactoryBean threw exception
>
> > > > on
>
> > >
>
> > > > object creation; nested exception is java.lang.NullPointerException:
>
> > >
>
> > > > Name is null
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.context.expression.StandardBeanExpressionResolver.
>
> > >
>
> > > > evaluate(StandardBeanExpressionResolver.java:164)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.eval
>
> > > > ua
>
> > > > te
>
> > >
>
> > > > BeanDefinitionString(AbstractBeanFactory.java:1448)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFacto
>
> > > > ry
>
> > > > .d
>
> > >
>
> > > > oResolveDependency(DefaultListableBeanFactory.java:1088)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFacto
>
> > > > ry
>
> > > > .r
>
> > >
>
> > > > esolveDependency(DefaultListableBeanFactory.java:1066)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanP
>
> > > > os
>
> > > > tP
>
> > >
>
> > > > rocessor.java:659)
>
> > >
>
> > > >
>
> > >
>
> > > > ... 48 common frames omitted
>
> > >
>
> > > >
>
> > >
>
> > > > Caused by:
>
> > >
>
> > > > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > >
>
> > > > Error creating bean with name
>
> > >
>
> > > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>
> > >
>
> > > > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > >
>
> > > > parameter 0; nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'jwtAuthenticationProvider' defined in
>
> > > > class
>
> > >
>
> > > > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > >
>
> > > > reference to bean 'authorizer' while setting constructor argument;
>
> > >
>
> > > > nested exception is
>
> > >
>
> > > > org.springframework.beans.factory.BeanCreationException: Error
>
> > >
>
> > > > creating bean with name 'authorizer': FactoryBean threw exception
>
> > > > on
>
> > >
>
> > > > object creation; nested exception is java.lang.NullPointerException:
>
> > >
>
> > > > Name is null
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanP
>
> > > > os
>
> > > > tP
>
> > >
>
> > > > rocessor.java:667)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.InjectionMetadata.inj
>
> > > > ec
>
> > > > t(
>
> > >
>
> > > > InjectionMetadata.java:88)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
>
> > > > an
>
> > > > Po
>
> > >
>
> > > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostP
>
> > > > ro
>
> > > > ce
>
> > >
>
> > > > ssor.java:366)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapableB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapableB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractAutowireCapableB
>
> > > > ea
>
> > > > nF
>
> > >
>
> > > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory$1.ge
>
> > > > tO
>
> > > > bj
>
> > >
>
> > > > ect(AbstractBeanFactory.java:306)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultSingletonBeanRegi
>
> > > > st
>
> > > > ry
>
> > >
>
> > > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.doGe
>
> > > > tB
>
> > > > ea
>
> > >
>
> > > > n(AbstractBeanFactory.java:302)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.AbstractBeanFactory.getB
>
> > > > ea
>
> > > > n(
>
> > >
>
> > > > AbstractBeanFactory.java:202)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFacto
>
> > > > ry
>
> > > > .g
>
> > >
>
> > > > etBeansOfType(DefaultListableBeanFactory.java:519)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.beans.factory.support.DefaultListableBeanFacto
>
> > > > ry
>
> > > > .g
>
> > >
>
> > > > etBeansOfType(DefaultListableBeanFactory.java:508)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.security.config.annotation.web.configuration.A
>
> > > > ut
>
> > > > ow
>
> > >
>
> > > > iredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(
>
> > > > Au
>
> > > > to
>
> > >
>
> > > > wiredWebSecurityConfigurersIgnoreParents.java:53)
>
> > >
>
> > > >
>
> > >
>
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>
> > > > Method)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm
>
> > > > pl
>
> > > > .j
>
> > >
>
> > > > ava:62)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc
>
> > > > ce
>
> > > > ss
>
> > >
>
> > > > orImpl.java:43)
>
> > >
>
> > > >
>
> > >
>
> > > > at java.lang.reflect.Method.invoke(Method.java:498)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.support.ReflectiveMethodExecut
>
> > > > or
>
> > > > .e
>
> > >
>
> > > > xecute(ReflectiveMethodExecutor.java:113)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.MethodReference.getValueIn
>
> > > > te
>
> > > > rn
>
> > >
>
> > > > al(MethodReference.java:129)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.MethodReference.access$000
>
> > > > (M
>
> > > > et
>
> > >
>
> > > > hodReference.java:49)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.MethodReference$MethodValu
>
> > > > eR
>
> > > > ef
>
> > >
>
> > > > .getValue(MethodReference.java:347)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.CompoundExpression.getValu
>
> > > > eI
>
> > > > nt
>
> > >
>
> > > > ernal(CompoundExpression.java:88)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.ast.SpelNodeImpl.getValue(Spel
>
> > > > No
>
> > > > de
>
> > >
>
> > > > Impl.java:120)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.expression.spel.standard.SpelExpression.getVal
>
> > > > ue
>
> > > > (S
>
> > >
>
> > > > pelExpression.java:262)
>
> > >
>
> > > >
>
> > >
>
> > > > at
>
> > >
>
> > > > org.springframework.context.expression.StandardBeanExpressionResolver.
>
> > >
>
> > > > evaluate(StandardBeanExpressionResolver.java:161)
>
> > >
>
> > > >
>
> > >
>
> > > > ... 52 common frames omitted
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > I tried to Google for possible clues, but so far, there hasn’t
>
> > > > been
>
> > >
>
> > > > any luck…
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > -----Original Message-----
>
> > >
>
> > > > From: Bryan Bende <bb...@gmail.com>
>
> > >
>
> > > > Sent: Monday, October 15, 2018 10:27 AM
>
> > >
>
> > > > To: users@nifi.apache.org
>
> > >
>
> > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > >
>
> > >
>
> > > > Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
>
> > >
>
> > > >
>
> > >
>
> > > > On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > -----Original Message-----
>
> > >
>
> > > >
>
> > >
>
> > > > > From: Bryan Bende <bb...@gmail.com>
>
> > >
>
> > > >
>
> > >
>
> > > > > Sent: Monday, October 15, 2018 9:43 AM
>
> > >
>
> > > >
>
> > >
>
> > > > > To: users@nifi.apache.org
>
> > >
>
> > > >
>
> > >
>
> > > > > Subject: Re: NiFi fails on cluster nodes
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
>
> > >
>
> > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > Mike
>
> > >
>
> > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > >
>
> > >
>
> > > > > > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >> Hi Mike and Bryan,
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >>
>
> > >
>
> > > >
>
> > >
>
> > > > > >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol
>
> > >
>
> > > > > >> Request-1]
>
> > >
>
> > > >
>
> > >
>
> > > > > >> o.a.nifi.security.util.CertificateUtils The incoming request
>
> > > > > >> did
>
> > >
>
> > > >
>
> > >
>
> > > > > >> not contain client certificates and thus the DN cannot be extracted.
>
> > >
>
> > > >
>
> > >
>
> > > > > >> Check that the other endp
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Hi Bryan,
Yes, converting two standalone NiFi instances into a cluster is exactly what we are trying to do. Here are the steps I went through in this round:
· restored the original configuration files (nifi.properties, users.xml, authorizers.xml and authorizations.xml)
· restarted one instance in the standalone mode
· added two new node users in the NiFi web UI (CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US and CN=<host-2, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US)
· granted them the “proxy user requests” privileges
· edited the nifi.properties file (nifi.state.management.embedded.zookeeper.start=true, nifi.cluster.is.node=true, nifi.zookeeper.connect.string=<host-1, redacted>:2181)
· restarted the node on host-1
On logging in, I see the cluster section of the dashboard showing 1/1 as expected, although I’m unable to do anything there due to errors like this:
Insufficient Permissions
Node <host-1, redacted>:8008 is unable to fulfill this request due to: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US Contact the system administrator.
The nifi-user.log also contains
2018-10-23 12:17:01,916 WARN [NiFi Web Server-224] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=<host-1, redacted>, OU=Devices, OU=NIH, OU=HHS, O=U.S. Government, C=US
From your experience, what the most likely causes for this exception?
Thank you,
Alexander
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Monday, October 22, 2018 1:25 PM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
Yes, to further clarify what I meant...
If you are trying to change the Initial Admin or Node Identities in authorizers.xml, these will only be used when there are no other users/group/policies present. People frequently make a mistake during initial config and then try to edit authorizers.xml and try again, but it won't actually do anything unless you remove the users.xml and authorizations.xml to start over.
In your case it sounds like you are trying to convert and existing standalone node to a cluster, given that I would do the following...
- In standalone mode, use the UI to add users for the DN's of the server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2, OU=NIFI)
- In the UI, grant those users Write access to "Proxy"
- Convert to a cluster and keep your same authorizers.xml, users.xml, and authorizations.xml when you setup your cluster, this way all your users and policies are already setup and the Initial Admin and Node Identities are not needed
On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Monday, October 22, 2018 12:48 PM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
> Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> This blog post shows how to setup a secure 2 node cluster:
>
> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-author
> ization-and-multi-tenancy
>
> The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> <authorizer>
> <identifier>file-provider</identifier>
> <class>org.apache.nifi.authorization.FileAuthorizer</class>
> <property name="Authorizations File">./conf/authorizations.xml</property>
> <property name="Users File">./conf/users.xml</property>
> <property name="Initial Admin Identity">CN=bbende, OU=ApacheNiFi</property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Node Identity 1">CN=localhost, OU=NIFI</property>
> </authorizer>
>
> You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> <userGroupProvider>
> <identifier>file-user-group-provider</identifier>
> <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
> <property name="Users File">./conf/users.xml</property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Initial User Identity 1">CN=bbende, OU=Apache NiFI</property>
> <property name="Initial User Identity 2">CN=nifi-host-1, OU=NIFI</property>
> <property name="Initial User Identity 2">CN=nifi-host-2, OU=NIFI</property>
> </userGroupProvider>
>
> <accessPolicyProvider>
> <identifier>file-access-policy-provider</identifier>
> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
> <property name="User Group
> Provider">composite-configurable-user-group-provider</property>
> <property name="Authorizations
> File">./conf/authorizations.xml</property>
> <property name="Initial Admin Identity">CN=bbende, OU=Apache NiFI</property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Node Identity 1">CN=nifi-host-1, OU=NIFI</property>
> <property name="Node Identity 1">CN=nifi-host-2, OU=NIFI</property>
> </accessPolicyProvider>
>
>
> Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > Hi Bryan,
> >
> >
> >
> > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
> >
> >
> >
> > Are you saying that we have to run ZooKeeper on both nodes? BTW, do
> > we still need
> >
> >
> >
> > nifi.login.identity.provider.configuration.file=./conf/login-identit
> > y-
> > providers.xml
> >
> >
> >
> > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
> >
> >
> >
> > Thank you,
> >
> >
> >
> > Alexander
> >
> >
> >
> > -----Original Message-----
> > From: Bryan Bende <bb...@gmail.com>>
> > Sent: Monday, October 22, 2018 11:55 AM
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
> > Subject: Re: NiFi fails on cluster nodes
> >
> >
> >
> > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
> >
> >
> >
> > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
> >
> >
> >
> > server.1=nifi-node1-hostname:2888:3888
> >
> > server.2=nifi-node2-hostname:2888:3888
> >
> > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > I wonder if anyone has run into the same problem when trying to
> >
> > > configure composite authentication/authorization (LDAP and local
> >
> > > file)? When we use the “stand-alone” authorizers.xml file with the
> >
> > > addition of two extra properties
> >
> > >
> >
> > >
> >
> > >
> >
> > > <property name="Node Identity 1">…
> >
> > >
> >
> > > <property name="Node Identity 2">…
> >
> > >
> >
> > >
> >
> > >
> >
> > > and let ZooKeeper start on one on the nodes, we end up with two
> >
> > > one-node clusters, since apparently, the NiFi instances don’t talk
> > > to
> >
> > > each other, but at least, they come alive…
> >
> > >
> >
> > >
> >
> > >
> >
> > > From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>>
> >
> > > Sent: Friday, October 19, 2018 11:18 AM
> >
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > > Subject: RE: NiFi fails on cluster nodes
> >
> > >
> >
> > >
> >
> > >
> >
> > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
> >
> > >
> >
> > >
> >
> > >
> >
> > > 2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader
> >
> > > Context initialization failed
> >
> > >
> >
> > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > > Error creating bean with name
> >
> > > 'org.springframework.security.config.annotation.web.configuration.
> > > We
> > > bS
> >
> > > ecurityConfiguration': Unsatisfied dependency expressed through
> > > method
> >
> > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
> > > exception
> >
> > > is org.springframework.beans.factory.BeanExpressionException:
> >
> > > Expression parsing failed; nested exception is
> >
> > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > > Error creating bean with name
> >
> > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> >
> > > dependency expressed through method 'setJwtAuthenticationProvider'
> >
> > > parameter 0; nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'jwtAuthenticationProvider' defined in
> > > class
> >
> > > path resource [nifi-web-security-context.xml]: Cannot resolve
> >
> > > reference to bean 'authorizer' while setting constructor argument;
> >
> > > nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'authorizer': FactoryBean threw exception
> > > on
> >
> > > object creation; nested exception is java.lang.NullPointerException:
> >
> > > Name is null
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
> > > an
> > > Po
> >
> > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanP
> > > os
> > > tP
> >
> > > rocessor.java:667)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.InjectionMetadata.inj
> > > ec
> > > t(
> >
> > > InjectionMetadata.java:88)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
> > > an
> > > Po
> >
> > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostP
> > > ro
> > > ce
> >
> > > ssor.java:366)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableB
> > > ea
> > > nF
> >
> > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableB
> > > ea
> > > nF
> >
> > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableB
> > > ea
> > > nF
> >
> > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory$1.ge
> > > tO
> > > bj
> >
> > > ect(AbstractBeanFactory.java:306)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultSingletonBeanRegi
> > > st
> > > ry
> >
> > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.doGe
> > > tB
> > > ea
> >
> > > n(AbstractBeanFactory.java:302)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.getB
> > > ea
> > > n(
> >
> > > AbstractBeanFactory.java:197)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFacto
> > > ry
> > > .p
> >
> > > reInstantiateSingletons(DefaultListableBeanFactory.java:761)
> >
> > >
> >
> > > at
> >
> > > org.springframework.context.support.AbstractApplicationContext.fin
> > > is
> > > hB
> >
> > > eanFactoryInitialization(AbstractApplicationContext.java:867)
> >
> > >
> >
> > > at
> >
> > > org.springframework.context.support.AbstractApplicationContext.ref
> > > re
> > > sh
> >
> > > (AbstractApplicationContext.java:543)
> >
> > >
> >
> > > at
> >
> > > org.springframework.web.context.ContextLoader.configureAndRefreshW
> > > eb
> > > Ap
> >
> > > plicationContext(ContextLoader.java:443)
> >
> > >
> >
> > > at
> >
> > > org.springframework.web.context.ContextLoader.initWebApplicationCo
> > > nt
> > > ex
> >
> > > t(ContextLoader.java:325)
> >
> > >
> >
> > > at
> >
> > > org.springframework.web.context.ContextLoaderListener.contextIniti
> > > al
> > > iz
> >
> > > ed(ContextLoaderListener.java:107)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.ContextHandler.callContextInitial
> > > iz
> > > ed
> >
> > > (ContextHandler.java:876)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitial
> > > iz
> > > ed
> >
> > > (ServletContextHandler.java:532)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.ContextHandler.startContext(Conte
> > > xt
> > > Ha
> >
> > > ndler.java:839)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(Servl
> > > et
> > > Co
> >
> > > ntextHandler.java:344)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:
> >
> > > 1480)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.
> > > ja
> > > va
> >
> > > :1442)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHan
> > > dl
> > > er
> >
> > > .java:799)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletCon
> > > te
> > > xt
> >
> > > Handler.java:261)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:
> > > 54
> > > 0)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
> > > if
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Containe
> > > rL
> > > if
> >
> > > eCycle.java:131)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Contai
> > > ne
> > > rL
> >
> > > ifeCycle.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractH
> > > an
> > > dl
> >
> > > er.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
> > > if
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Containe
> > > rL
> > > if
> >
> > > eCycle.java:131)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Contai
> > > ne
> > > rL
> >
> > > ifeCycle.java:105)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractH
> > > an
> > > dl
> >
> > > er.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.
> >
> > > java:290)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
> > > if
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Containe
> > > rL
> > > if
> >
> > > eCycle.java:131)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Contai
> > > ne
> > > rL
> >
> > > ifeCycle.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractH
> > > an
> > > dl
> >
> > > er.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
> > > if
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(Containe
> > > rL
> > > if
> >
> > > eCycle.java:131)
> >
> > >
> >
> > > at org.eclipse.jetty.server.Server.start(Server.java:452)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Contai
> > > ne
> > > rL
> >
> > > ifeCycle.java:105)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractH
> > > an
> > > dl
> >
> > > er.java:113)
> >
> > >
> >
> > > at
> > > org.eclipse.jetty.server.Server.doStart(Server.java:419)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractL
> > > if
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838)
> >
> > >
> >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:157)
> >
> > >
> >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> >
> > >
> >
> > > at org.apache.nifi.NiFi.main(NiFi.java:292)
> >
> > >
> >
> > > Caused by: org.springframework.beans.factory.BeanExpressionException:
> >
> > > Expression parsing failed; nested exception is
> >
> > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > > Error creating bean with name
> >
> > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> >
> > > dependency expressed through method 'setJwtAuthenticationProvider'
> >
> > > parameter 0; nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'jwtAuthenticationProvider' defined in
> > > class
> >
> > > path resource [nifi-web-security-context.xml]: Cannot resolve
> >
> > > reference to bean 'authorizer' while setting constructor argument;
> >
> > > nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'authorizer': FactoryBean threw exception
> > > on
> >
> > > object creation; nested exception is java.lang.NullPointerException:
> >
> > > Name is null
> >
> > >
> >
> > > at
> >
> > > org.springframework.context.expression.StandardBeanExpressionResolver.
> >
> > > evaluate(StandardBeanExpressionResolver.java:164)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.eval
> > > ua
> > > te
> >
> > > BeanDefinitionString(AbstractBeanFactory.java:1448)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFacto
> > > ry
> > > .d
> >
> > > oResolveDependency(DefaultListableBeanFactory.java:1088)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFacto
> > > ry
> > > .r
> >
> > > esolveDependency(DefaultListableBeanFactory.java:1066)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
> > > an
> > > Po
> >
> > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanP
> > > os
> > > tP
> >
> > > rocessor.java:659)
> >
> > >
> >
> > > ... 48 common frames omitted
> >
> > >
> >
> > > Caused by:
> >
> > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > > Error creating bean with name
> >
> > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> >
> > > dependency expressed through method 'setJwtAuthenticationProvider'
> >
> > > parameter 0; nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'jwtAuthenticationProvider' defined in
> > > class
> >
> > > path resource [nifi-web-security-context.xml]: Cannot resolve
> >
> > > reference to bean 'authorizer' while setting constructor argument;
> >
> > > nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'authorizer': FactoryBean threw exception
> > > on
> >
> > > object creation; nested exception is java.lang.NullPointerException:
> >
> > > Name is null
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
> > > an
> > > Po
> >
> > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanP
> > > os
> > > tP
> >
> > > rocessor.java:667)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.InjectionMetadata.inj
> > > ec
> > > t(
> >
> > > InjectionMetadata.java:88)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBe
> > > an
> > > Po
> >
> > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostP
> > > ro
> > > ce
> >
> > > ssor.java:366)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableB
> > > ea
> > > nF
> >
> > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableB
> > > ea
> > > nF
> >
> > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableB
> > > ea
> > > nF
> >
> > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory$1.ge
> > > tO
> > > bj
> >
> > > ect(AbstractBeanFactory.java:306)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultSingletonBeanRegi
> > > st
> > > ry
> >
> > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.doGe
> > > tB
> > > ea
> >
> > > n(AbstractBeanFactory.java:302)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.getB
> > > ea
> > > n(
> >
> > > AbstractBeanFactory.java:202)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFacto
> > > ry
> > > .g
> >
> > > etBeansOfType(DefaultListableBeanFactory.java:519)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFacto
> > > ry
> > > .g
> >
> > > etBeansOfType(DefaultListableBeanFactory.java:508)
> >
> > >
> >
> > > at
> >
> > > org.springframework.security.config.annotation.web.configuration.A
> > > ut
> > > ow
> >
> > > iredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(
> > > Au
> > > to
> >
> > > wiredWebSecurityConfigurersIgnoreParents.java:53)
> >
> > >
> >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method)
> >
> > >
> >
> > > at
> >
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm
> > > pl
> > > .j
> >
> > > ava:62)
> >
> > >
> >
> > > at
> >
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc
> > > ce
> > > ss
> >
> > > orImpl.java:43)
> >
> > >
> >
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.support.ReflectiveMethodExecut
> > > or
> > > .e
> >
> > > xecute(ReflectiveMethodExecutor.java:113)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.MethodReference.getValueIn
> > > te
> > > rn
> >
> > > al(MethodReference.java:129)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.MethodReference.access$000
> > > (M
> > > et
> >
> > > hodReference.java:49)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.MethodReference$MethodValu
> > > eR
> > > ef
> >
> > > .getValue(MethodReference.java:347)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.CompoundExpression.getValu
> > > eI
> > > nt
> >
> > > ernal(CompoundExpression.java:88)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.SpelNodeImpl.getValue(Spel
> > > No
> > > de
> >
> > > Impl.java:120)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.standard.SpelExpression.getVal
> > > ue
> > > (S
> >
> > > pelExpression.java:262)
> >
> > >
> >
> > > at
> >
> > > org.springframework.context.expression.StandardBeanExpressionResolver.
> >
> > > evaluate(StandardBeanExpressionResolver.java:161)
> >
> > >
> >
> > > ... 52 common frames omitted
> >
> > >
> >
> > >
> >
> > >
> >
> > > I tried to Google for possible clues, but so far, there hasn’t
> > > been
> >
> > > any luck…
> >
> > >
> >
> > >
> >
> > >
> >
> > > -----Original Message-----
> >
> > > From: Bryan Bende <bb...@gmail.com>>
> >
> > > Sent: Monday, October 15, 2018 10:27 AM
> >
> > > To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > >
> >
> > >
> >
> > > I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
> >
> > >
> >
> > >
> >
> > >
> >
> > > In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
> >
> > >
> >
> > >
> >
> > >
> >
> > > Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
> >
> > >
> >
> > > On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > -----Original Message-----
> >
> > >
> >
> > > > From: Bryan Bende <bb...@gmail.com>>
> >
> > >
> >
> > > > Sent: Monday, October 15, 2018 9:43 AM
> >
> > >
> >
> > > > To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com>> wrote:
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > Mike
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Hi Mike and Bryan,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol
> >
> > > > >> Request-1]
> >
> > >
> >
> > > > >> o.a.nifi.security.util.CertificateUtils The incoming request
> > > > >> did
> >
> > >
> >
> > > > >> not contain client certificates and thus the DN cannot be extracted.
> >
> > >
> >
> > > > >> Check that the other endpoint is providing a complete client
> >
> > >
> >
> > > > >> certificate chain
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-15 09:19:48,425 INFO [main]
> >
> > >
> >
> > > > >> o.a.nifi.controller.StandardFlowService Connecting Node:
> >
> > >
> >
> > > > >> 0.0.0.0:8008
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol
> >
> > > > >> Request-2]
> >
> > >
> >
> > > > >> o.a.nifi.security.util.CertificateUtils The incoming request
> > > > >> did
> >
> > >
> >
> > > > >> not contain client certificates and thus the DN cannot be extracted.
> >
> > >
> >
> > > > >> Check that the other endpoint is providing a complete client
> >
> > >
> >
> > > > >> certificate chain
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-15 09:19:48,456 WARN [main]
> >
> > >
> >
> > > > >> o.a.nifi.controller.StandardFlowService Failed to connect to
> >
> > >
> >
> > > > >> cluster due to: org.apache.nifi.cluster.protocol.ProtocolException:
> >
> > >
> >
> > > > >> Failed marshalling 'CONNECTION_REQUEST' protocol message due to:
> >
> > >
> >
> > > > >> javax.net.ssl.SSLHandshakeException: Received fatal alert:
> >
> > >
> >
> > > > >> bad_certificate
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Thank you.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: Mike Thomsen <mi...@gmail.com>>
> >
> > >
> >
> > > > >> Sent: Monday, October 15, 2018 9:02 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> http://nifi.apache.org/docs/nifi-docs/html/administration-gui
> > > > >> de
> > > > >> .h
> >
> > > > >> tm
> >
> > >
> >
> > > > >> l
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> See the properties that start with "nifi.zookeeper."
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Mike,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Thanks again.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: Mike Thomsen <mi...@gmail.com>>
> >
> > >
> >
> > > > >> Sent: Friday, October 12, 2018 10:17 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 08:21:35,838 WARN [main]
> >
> > >
> >
> > > > >> o.a.nifi.controller.StandardFlowService There is currently no
> >
> > >
> >
> > > > >> Cluster Coordinator. This often happens upon restart of NiFi
> > > > >> when
> >
> > >
> >
> > > > >> running an embedded ZooKeeper. Will register this node to
> > > > >> become
> >
> > >
> >
> > > > >> the active Cluster Coordinator and will attempt to connect to
> >
> > >
> >
> > > > >> cluster again
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 08:21:35,838 INFO [main]
> >
> > >
> >
> > > > >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >
> > >
> >
> > > > >> CuratorLeaderElectionManager[stopped=false] Attempted to
> > > > >> register
> >
> > >
> >
> > > > >> Leader Election for role 'Cluster Coordinator' but this role
> > > > >> is
> >
> > >
> >
> > > > >> already registered
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
> >
> > >
> >
> > > > >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 08:21:42,092 INFO
> > > > >> [Curator-ConnectionStateManager-0]
> >
> > >
> >
> > > > >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >
> > >
> >
> > > > >> org.apache.nifi.controller.leader.election.CuratorLeaderElect
> > > > >> io
> > > > >> nM
> >
> > > > >> an
> >
> > >
> >
> > > > >> ag er$ElectionListener@17900f5b Connection State changed to
> >
> > >
> >
> > > > >> SUSPENDED
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: Mike Thomsen <mi...@gmail.com>>
> >
> > >
> >
> > > > >> Sent: Friday, October 12, 2018 8:33 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Alexander,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> I am pretty sure your problem is here:
> >
> > >
> >
> > > > >> nifi.state.management.embedded.zookeeper.start=true
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> What you need to try is these steps:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 1. Set up an external ZooKeeper instance (or set up 3 in a
> >
> > > > >> quorum;
> >
> > >
> >
> > > > >> must be odd numbers)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 3. Restart all of them.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> See if that works.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Mike
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
> >
> > >
> >
> > > > >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry
> > > > >> gave
> >
> > >
> >
> > > > >> up
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >
> > >
> >
> > > > >> KeeperErrorCode = ConnectionLoss
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:
> >
> > > > >> 99
> >
> > >
> >
> > > > >> )
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkB
> > > > >> ac
> > > > >> kg
> >
> > > > >> ro
> >
> > >
> >
> > > > >> un
> >
> > >
> >
> > > > >> dRetry(CuratorFrameworkImpl.java:728)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.perfor
> > > > >> mB
> > > > >> ac
> >
> > > > >> kg
> >
> > >
> >
> > > > >> ro
> >
> > >
> >
> > > > >> undOperation(CuratorFrameworkImpl.java:857)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgr
> > > > >> ou
> > > > >> nd
> >
> > > > >> Op
> >
> > >
> >
> > > > >> er
> >
> > >
> >
> > > > >> ationsLoop(CuratorFrameworkImpl.java:809)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access
> > > > >> $3
> > > > >> 00
> >
> > > > >> (C
> >
> > >
> >
> > > > >> ur
> >
> > >
> >
> > > > >> atorFrameworkImpl.java:64)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call
> > > > >> (C
> > > > >> ur
> >
> > > > >> at
> >
> > >
> >
> > > > >> or
> >
> > >
> >
> > > > >> FrameworkImpl.java:267)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > > > >> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >
> > >
> >
> > > > >> access$201(ScheduledThreadPoolExecutor.java:180)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >
> > >
> >
> > > > >> run(ScheduledThreadPoolExecutor.java:293)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> >
> > >
> >
> > > > >> java:1149)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPool
> > > > >> Ex
> > > > >> ec
> >
> > > > >> ut
> >
> > >
> >
> > > > >> or
> >
> > >
> >
> > > > >> .java:624)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at java.lang.Thread.run(Thread.java:748)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Is there anything else we should check?
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: Nathan Gough <th...@gmail.com>>
> >
> > >
> >
> > > > >> Sent: Thursday, October 11, 2018 9:12 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: ashmeet kandhari <as...@gmail.com>>
> >
> > >
> >
> > > > >> Reply-To: <us...@nifi.apache.org>>
> >
> > >
> >
> > > > >> Date: Thursday, October 11, 2018 at 9:09 AM
> >
> > >
> >
> > > > >> To: <us...@nifi.apache.org>>
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Hi Alexander,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: ashmeet kandhari <as...@gmail.com>>
> >
> > >
> >
> > > > >> Sent: Thursday, October 11, 2018 4:36 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Can you see if zookeeper node is up and running and can
> > > > >> connect
> >
> > > > >> to
> >
> > >
> >
> > > > >> the nifi nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Hello,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,748 INFO [main]
> >
> > >
> >
> > > > >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125
> > > > >> properties
> >
> > > > >> from
> >
> > >
> >
> > > > >> /opt/nifi-1.7.1/./conf/nifi.properties
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi
> > > > >> Loaded
> >
> > > > >> 125
> >
> > >
> >
> > > > >> properties
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,762 INFO [main]
> >
> > >
> >
> > > > >> org.apache.nifi.BootstrapListener Started Bootstrap Listener,
> >
> > >
> >
> > > > >> Listening for incoming requests on port
> >
> > >
> >
> > > > >> 43744
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi
> > > > >> Failure
> >
> > >
> >
> > > > >> to launch NiFi due to java.net.ConnectException: Connection
> > > > >> timed
> >
> > >
> >
> > > > >> out (Connection timed out)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> java.net.ConnectException: Connection timed out (Connection
> > > > >> timed
> >
> > >
> >
> > > > >> out)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at java.net.PlainSocketImpl.socketConnect(Native
> > > > >> Method)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.
> >
> > >
> >
> > > > >> ja
> >
> > >
> >
> > > > >> va:350)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPla
> > > > >> in
> > > > >> So
> >
> > > > >> ck
> >
> > >
> >
> > > > >> et
> >
> > >
> >
> > > > >> Impl.java:206)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.
> >
> > > > >> ja
> >
> > >
> >
> > > > >> va
> >
> > >
> >
> > > > >> :188)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at java.net.Socket.connect(Socket.java:589)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at java.net.Socket.connect(Socket.java:538)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
> >
> > >
> >
> > > > >> 100)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:
> > > > >> 83
> >
> > > > >> )
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at org.apache.nifi.NiFi.main(NiFi.java:292)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Thanks in advance,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Alexander
>
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
Yes, to further clarify what I meant...
If you are trying to change the Initial Admin or Node Identities in
authorizers.xml, these will only be used when there are no other
users/group/policies present. People frequently make a mistake during
initial config and then try to edit authorizers.xml and try again, but
it won't actually do anything unless you remove the users.xml and
authorizations.xml to start over.
In your case it sounds like you are trying to convert and existing
standalone node to a cluster, given that I would do the following...
- In standalone mode, use the UI to add users for the DN's of the
server certificates (CN=nifi-node-1, OU=NIFI, CN=nifi-node-2, OU=NIFI)
- In the UI, grant those users Write access to "Proxy"
- Convert to a cluster and keep your same authorizers.xml, users.xml,
and authorizations.xml when you setup your cluster, this way all your
users and policies are already setup and the Initial Admin and Node
Identities are not needed
On Mon, Oct 22, 2018 at 1:06 PM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Monday, October 22, 2018 12:48 PM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
> Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
>
> You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
>
> This blog post shows how to setup a secure 2 node cluster:
>
> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>
> The only difference is that the authorizers.xml has changed slightly, so instead of:
>
> <authorizer>
> <identifier>file-provider</identifier>
> <class>org.apache.nifi.authorization.FileAuthorizer</class>
> <property name="Authorizations File">./conf/authorizations.xml</property>
> <property name="Users File">./conf/users.xml</property>
> <property name="Initial Admin Identity">CN=bbende, OU=ApacheNiFi</property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Node Identity 1">CN=localhost, OU=NIFI</property> </authorizer>
>
> You need to add the the users to the user-group-provider and then to the access-policy-provider...
>
> <userGroupProvider>
> <identifier>file-user-group-provider</identifier>
> <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
> <property name="Users File">./conf/users.xml</property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Initial User Identity 1">CN=bbende, OU=Apache NiFI</property>
> <property name="Initial User Identity 2">CN=nifi-host-1, OU=NIFI</property>
> <property name="Initial User Identity 2">CN=nifi-host-2, OU=NIFI</property>
> </userGroupProvider>
>
> <accessPolicyProvider>
> <identifier>file-access-policy-provider</identifier>
> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
> <property name="User Group
> Provider">composite-configurable-user-group-provider</property>
> <property name="Authorizations
> File">./conf/authorizations.xml</property>
> <property name="Initial Admin Identity">CN=bbende, OU=Apache NiFI</property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Node Identity 1">CN=nifi-host-1, OU=NIFI</property>
> <property name="Node Identity 1">CN=nifi-host-2, OU=NIFI</property>
> </accessPolicyProvider>
>
>
> Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > Hi Bryan,
> >
> >
> >
> > At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
> >
> >
> >
> > Are you saying that we have to run ZooKeeper on both nodes? BTW, do we
> > still need
> >
> >
> >
> > nifi.login.identity.provider.configuration.file=./conf/login-identity-
> > providers.xml
> >
> >
> >
> > in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
> >
> >
> >
> > Thank you,
> >
> >
> >
> > Alexander
> >
> >
> >
> > -----Original Message-----
> > From: Bryan Bende <bb...@gmail.com>
> > Sent: Monday, October 22, 2018 11:55 AM
> > To: users@nifi.apache.org
> > Subject: Re: NiFi fails on cluster nodes
> >
> >
> >
> > If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
> >
> >
> >
> > In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
> >
> >
> >
> > server.1=nifi-node1-hostname:2888:3888
> >
> > server.2=nifi-node2-hostname:2888:3888
> >
> > On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > I wonder if anyone has run into the same problem when trying to
> >
> > > configure composite authentication/authorization (LDAP and local
> >
> > > file)? When we use the “stand-alone” authorizers.xml file with the
> >
> > > addition of two extra properties
> >
> > >
> >
> > >
> >
> > >
> >
> > > <property name="Node Identity 1">…
> >
> > >
> >
> > > <property name="Node Identity 2">…
> >
> > >
> >
> > >
> >
> > >
> >
> > > and let ZooKeeper start on one on the nodes, we end up with two
> >
> > > one-node clusters, since apparently, the NiFi instances don’t talk
> > > to
> >
> > > each other, but at least, they come alive…
> >
> > >
> >
> > >
> >
> > >
> >
> > > From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
> >
> > > Sent: Friday, October 19, 2018 11:18 AM
> >
> > > To: users@nifi.apache.org
> >
> > > Subject: RE: NiFi fails on cluster nodes
> >
> > >
> >
> > >
> >
> > >
> >
> > > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
> >
> > >
> >
> > >
> >
> > >
> >
> > > 2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader
> >
> > > Context initialization failed
> >
> > >
> >
> > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > > Error creating bean with name
> >
> > > 'org.springframework.security.config.annotation.web.configuration.We
> > > bS
> >
> > > ecurityConfiguration': Unsatisfied dependency expressed through
> > > method
> >
> > > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
> > > exception
> >
> > > is org.springframework.beans.factory.BeanExpressionException:
> >
> > > Expression parsing failed; nested exception is
> >
> > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > > Error creating bean with name
> >
> > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> >
> > > dependency expressed through method 'setJwtAuthenticationProvider'
> >
> > > parameter 0; nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'jwtAuthenticationProvider' defined in class
> >
> > > path resource [nifi-web-security-context.xml]: Cannot resolve
> >
> > > reference to bean 'authorizer' while setting constructor argument;
> >
> > > nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'authorizer': FactoryBean threw exception on
> >
> > > object creation; nested exception is java.lang.NullPointerException:
> >
> > > Name is null
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > > Po
> >
> > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPos
> > > tP
> >
> > > rocessor.java:667)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.InjectionMetadata.injec
> > > t(
> >
> > > InjectionMetadata.java:88)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > > Po
> >
> > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostPro
> > > ce
> >
> > > ssor.java:366)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > > nF
> >
> > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > > nF
> >
> > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > > nF
> >
> > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory$1.getO
> > > bj
> >
> > > ect(AbstractBeanFactory.java:306)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultSingletonBeanRegist
> > > ry
> >
> > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.doGetB
> > > ea
> >
> > > n(AbstractBeanFactory.java:302)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.getBea
> > > n(
> >
> > > AbstractBeanFactory.java:197)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > > .p
> >
> > > reInstantiateSingletons(DefaultListableBeanFactory.java:761)
> >
> > >
> >
> > > at
> >
> > > org.springframework.context.support.AbstractApplicationContext.finis
> > > hB
> >
> > > eanFactoryInitialization(AbstractApplicationContext.java:867)
> >
> > >
> >
> > > at
> >
> > > org.springframework.context.support.AbstractApplicationContext.refre
> > > sh
> >
> > > (AbstractApplicationContext.java:543)
> >
> > >
> >
> > > at
> >
> > > org.springframework.web.context.ContextLoader.configureAndRefreshWeb
> > > Ap
> >
> > > plicationContext(ContextLoader.java:443)
> >
> > >
> >
> > > at
> >
> > > org.springframework.web.context.ContextLoader.initWebApplicationCont
> > > ex
> >
> > > t(ContextLoader.java:325)
> >
> > >
> >
> > > at
> >
> > > org.springframework.web.context.ContextLoaderListener.contextInitial
> > > iz
> >
> > > ed(ContextLoaderListener.java:107)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.ContextHandler.callContextInitializ
> > > ed
> >
> > > (ContextHandler.java:876)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitializ
> > > ed
> >
> > > (ServletContextHandler.java:532)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.ContextHandler.startContext(Context
> > > Ha
> >
> > > ndler.java:839)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(Servlet
> > > Co
> >
> > > ntextHandler.java:344)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:
> >
> > > 1480)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.ja
> > > va
> >
> > > :1442)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandl
> > > er
> >
> > > .java:799)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletConte
> > > xt
> >
> > > Handler.java:261)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:54
> > > 0)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL
> > > if
> >
> > > eCycle.java:131)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe
> > > rL
> >
> > > ifeCycle.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan
> > > dl
> >
> > > er.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL
> > > if
> >
> > > eCycle.java:131)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe
> > > rL
> >
> > > ifeCycle.java:105)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan
> > > dl
> >
> > > er.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.
> >
> > > java:290)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL
> > > if
> >
> > > eCycle.java:131)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe
> > > rL
> >
> > > ifeCycle.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan
> > > dl
> >
> > > er.java:113)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL
> > > if
> >
> > > eCycle.java:131)
> >
> > >
> >
> > > at org.eclipse.jetty.server.Server.start(Server.java:452)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe
> > > rL
> >
> > > ifeCycle.java:105)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan
> > > dl
> >
> > > er.java:113)
> >
> > >
> >
> > > at org.eclipse.jetty.server.Server.doStart(Server.java:419)
> >
> > >
> >
> > > at
> >
> > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > > eC
> >
> > > ycle.java:68)
> >
> > >
> >
> > > at
> >
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838)
> >
> > >
> >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:157)
> >
> > >
> >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> >
> > >
> >
> > > at org.apache.nifi.NiFi.main(NiFi.java:292)
> >
> > >
> >
> > > Caused by: org.springframework.beans.factory.BeanExpressionException:
> >
> > > Expression parsing failed; nested exception is
> >
> > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > > Error creating bean with name
> >
> > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> >
> > > dependency expressed through method 'setJwtAuthenticationProvider'
> >
> > > parameter 0; nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'jwtAuthenticationProvider' defined in class
> >
> > > path resource [nifi-web-security-context.xml]: Cannot resolve
> >
> > > reference to bean 'authorizer' while setting constructor argument;
> >
> > > nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'authorizer': FactoryBean threw exception on
> >
> > > object creation; nested exception is java.lang.NullPointerException:
> >
> > > Name is null
> >
> > >
> >
> > > at
> >
> > > org.springframework.context.expression.StandardBeanExpressionResolver.
> >
> > > evaluate(StandardBeanExpressionResolver.java:164)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.evalua
> > > te
> >
> > > BeanDefinitionString(AbstractBeanFactory.java:1448)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > > .d
> >
> > > oResolveDependency(DefaultListableBeanFactory.java:1088)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > > .r
> >
> > > esolveDependency(DefaultListableBeanFactory.java:1066)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > > Po
> >
> > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPos
> > > tP
> >
> > > rocessor.java:659)
> >
> > >
> >
> > > ... 48 common frames omitted
> >
> > >
> >
> > > Caused by:
> >
> > > org.springframework.beans.factory.UnsatisfiedDependencyException:
> >
> > > Error creating bean with name
> >
> > > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> >
> > > dependency expressed through method 'setJwtAuthenticationProvider'
> >
> > > parameter 0; nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'jwtAuthenticationProvider' defined in class
> >
> > > path resource [nifi-web-security-context.xml]: Cannot resolve
> >
> > > reference to bean 'authorizer' while setting constructor argument;
> >
> > > nested exception is
> >
> > > org.springframework.beans.factory.BeanCreationException: Error
> >
> > > creating bean with name 'authorizer': FactoryBean threw exception on
> >
> > > object creation; nested exception is java.lang.NullPointerException:
> >
> > > Name is null
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > > Po
> >
> > > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPos
> > > tP
> >
> > > rocessor.java:667)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.InjectionMetadata.injec
> > > t(
> >
> > > InjectionMetadata.java:88)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > > Po
> >
> > > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostPro
> > > ce
> >
> > > ssor.java:366)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > > nF
> >
> > > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > > nF
> >
> > > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > > nF
> >
> > > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory$1.getO
> > > bj
> >
> > > ect(AbstractBeanFactory.java:306)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultSingletonBeanRegist
> > > ry
> >
> > > .getSingleton(DefaultSingletonBeanRegistry.java:230)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.doGetB
> > > ea
> >
> > > n(AbstractBeanFactory.java:302)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.AbstractBeanFactory.getBea
> > > n(
> >
> > > AbstractBeanFactory.java:202)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > > .g
> >
> > > etBeansOfType(DefaultListableBeanFactory.java:519)
> >
> > >
> >
> > > at
> >
> > > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > > .g
> >
> > > etBeansOfType(DefaultListableBeanFactory.java:508)
> >
> > >
> >
> > > at
> >
> > > org.springframework.security.config.annotation.web.configuration.Aut
> > > ow
> >
> > > iredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(Au
> > > to
> >
> > > wiredWebSecurityConfigurersIgnoreParents.java:53)
> >
> > >
> >
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method)
> >
> > >
> >
> > > at
> >
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
> > > .j
> >
> > > ava:62)
> >
> > >
> >
> > > at
> >
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
> > > ss
> >
> > > orImpl.java:43)
> >
> > >
> >
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.support.ReflectiveMethodExecutor
> > > .e
> >
> > > xecute(ReflectiveMethodExecutor.java:113)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.MethodReference.getValueInte
> > > rn
> >
> > > al(MethodReference.java:129)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.MethodReference.access$000(M
> > > et
> >
> > > hodReference.java:49)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.MethodReference$MethodValueR
> > > ef
> >
> > > .getValue(MethodReference.java:347)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.CompoundExpression.getValueI
> > > nt
> >
> > > ernal(CompoundExpression.java:88)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNo
> > > de
> >
> > > Impl.java:120)
> >
> > >
> >
> > > at
> >
> > > org.springframework.expression.spel.standard.SpelExpression.getValue
> > > (S
> >
> > > pelExpression.java:262)
> >
> > >
> >
> > > at
> >
> > > org.springframework.context.expression.StandardBeanExpressionResolver.
> >
> > > evaluate(StandardBeanExpressionResolver.java:161)
> >
> > >
> >
> > > ... 52 common frames omitted
> >
> > >
> >
> > >
> >
> > >
> >
> > > I tried to Google for possible clues, but so far, there hasn’t been
> >
> > > any luck…
> >
> > >
> >
> > >
> >
> > >
> >
> > > -----Original Message-----
> >
> > > From: Bryan Bende <bb...@gmail.com>
> >
> > > Sent: Monday, October 15, 2018 10:27 AM
> >
> > > To: users@nifi.apache.org
> >
> > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > >
> >
> > >
> >
> > > I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
> >
> > >
> >
> > >
> >
> > >
> >
> > > In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
> >
> > >
> >
> > >
> >
> > >
> >
> > > Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
> >
> > >
> >
> > > On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > -----Original Message-----
> >
> > >
> >
> > > > From: Bryan Bende <bb...@gmail.com>
> >
> > >
> >
> > > > Sent: Monday, October 15, 2018 9:43 AM
> >
> > >
> >
> > > > To: users@nifi.apache.org
> >
> > >
> >
> > > > Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
> >
> > >
> >
> > > >
> >
> > >
> >
> > > > On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com> wrote:
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > Mike
> >
> > >
> >
> > > > >
> >
> > >
> >
> > > > > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Hi Mike and Bryan,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol
> >
> > > > >> Request-1]
> >
> > >
> >
> > > > >> o.a.nifi.security.util.CertificateUtils The incoming request
> > > > >> did
> >
> > >
> >
> > > > >> not contain client certificates and thus the DN cannot be extracted.
> >
> > >
> >
> > > > >> Check that the other endpoint is providing a complete client
> >
> > >
> >
> > > > >> certificate chain
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-15 09:19:48,425 INFO [main]
> >
> > >
> >
> > > > >> o.a.nifi.controller.StandardFlowService Connecting Node:
> >
> > >
> >
> > > > >> 0.0.0.0:8008
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol
> >
> > > > >> Request-2]
> >
> > >
> >
> > > > >> o.a.nifi.security.util.CertificateUtils The incoming request
> > > > >> did
> >
> > >
> >
> > > > >> not contain client certificates and thus the DN cannot be extracted.
> >
> > >
> >
> > > > >> Check that the other endpoint is providing a complete client
> >
> > >
> >
> > > > >> certificate chain
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-15 09:19:48,456 WARN [main]
> >
> > >
> >
> > > > >> o.a.nifi.controller.StandardFlowService Failed to connect to
> >
> > >
> >
> > > > >> cluster due to: org.apache.nifi.cluster.protocol.ProtocolException:
> >
> > >
> >
> > > > >> Failed marshalling 'CONNECTION_REQUEST' protocol message due to:
> >
> > >
> >
> > > > >> javax.net.ssl.SSLHandshakeException: Received fatal alert:
> >
> > >
> >
> > > > >> bad_certificate
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Thank you.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: Mike Thomsen <mi...@gmail.com>
> >
> > >
> >
> > > > >> Sent: Monday, October 15, 2018 9:02 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> http://nifi.apache.org/docs/nifi-docs/html/administration-guide
> > > > >> .h
> >
> > > > >> tm
> >
> > >
> >
> > > > >> l
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> See the properties that start with "nifi.zookeeper."
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Mike,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Thanks again.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: Mike Thomsen <mi...@gmail.com>
> >
> > >
> >
> > > > >> Sent: Friday, October 12, 2018 10:17 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 08:21:35,838 WARN [main]
> >
> > >
> >
> > > > >> o.a.nifi.controller.StandardFlowService There is currently no
> >
> > >
> >
> > > > >> Cluster Coordinator. This often happens upon restart of NiFi
> > > > >> when
> >
> > >
> >
> > > > >> running an embedded ZooKeeper. Will register this node to
> > > > >> become
> >
> > >
> >
> > > > >> the active Cluster Coordinator and will attempt to connect to
> >
> > >
> >
> > > > >> cluster again
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 08:21:35,838 INFO [main]
> >
> > >
> >
> > > > >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >
> > >
> >
> > > > >> CuratorLeaderElectionManager[stopped=false] Attempted to
> > > > >> register
> >
> > >
> >
> > > > >> Leader Election for role 'Cluster Coordinator' but this role is
> >
> > >
> >
> > > > >> already registered
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
> >
> > >
> >
> > > > >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
> >
> > >
> >
> > > > >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >
> > >
> >
> > > > >> org.apache.nifi.controller.leader.election.CuratorLeaderElectio
> > > > >> nM
> >
> > > > >> an
> >
> > >
> >
> > > > >> ag er$ElectionListener@17900f5b Connection State changed to
> >
> > >
> >
> > > > >> SUSPENDED
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: Mike Thomsen <mi...@gmail.com>
> >
> > >
> >
> > > > >> Sent: Friday, October 12, 2018 8:33 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Alexander,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> I am pretty sure your problem is here:
> >
> > >
> >
> > > > >> nifi.state.management.embedded.zookeeper.start=true
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> What you need to try is these steps:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 1. Set up an external ZooKeeper instance (or set up 3 in a
> >
> > > > >> quorum;
> >
> > >
> >
> > > > >> must be odd numbers)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 3. Restart all of them.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> See if that works.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Mike
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
> >
> > >
> >
> > > > >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry
> > > > >> gave
> >
> > >
> >
> > > > >> up
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >
> > >
> >
> > > > >> KeeperErrorCode = ConnectionLoss
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:
> >
> > > > >> 99
> >
> > >
> >
> > > > >> )
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBac
> > > > >> kg
> >
> > > > >> ro
> >
> > >
> >
> > > > >> un
> >
> > >
> >
> > > > >> dRetry(CuratorFrameworkImpl.java:728)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.performB
> > > > >> ac
> >
> > > > >> kg
> >
> > >
> >
> > > > >> ro
> >
> > >
> >
> > > > >> undOperation(CuratorFrameworkImpl.java:857)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgrou
> > > > >> nd
> >
> > > > >> Op
> >
> > >
> >
> > > > >> er
> >
> > >
> >
> > > > >> ationsLoop(CuratorFrameworkImpl.java:809)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$3
> > > > >> 00
> >
> > > > >> (C
> >
> > >
> >
> > > > >> ur
> >
> > >
> >
> > > > >> atorFrameworkImpl.java:64)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(C
> > > > >> ur
> >
> > > > >> at
> >
> > >
> >
> > > > >> or
> >
> > >
> >
> > > > >> FrameworkImpl.java:267)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > > > >> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >
> > >
> >
> > > > >> access$201(ScheduledThreadPoolExecutor.java:180)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >
> > >
> >
> > > > >> run(ScheduledThreadPoolExecutor.java:293)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> >
> > >
> >
> > > > >> java:1149)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx
> > > > >> ec
> >
> > > > >> ut
> >
> > >
> >
> > > > >> or
> >
> > >
> >
> > > > >> .java:624)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at java.lang.Thread.run(Thread.java:748)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Is there anything else we should check?
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: Nathan Gough <th...@gmail.com>
> >
> > >
> >
> > > > >> Sent: Thursday, October 11, 2018 9:12 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: ashmeet kandhari <as...@gmail.com>
> >
> > >
> >
> > > > >> Reply-To: <us...@nifi.apache.org>
> >
> > >
> >
> > > > >> Date: Thursday, October 11, 2018 at 9:09 AM
> >
> > >
> >
> > > > >> To: <us...@nifi.apache.org>
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Hi Alexander,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> From: ashmeet kandhari <as...@gmail.com>
> >
> > >
> >
> > > > >> Sent: Thursday, October 11, 2018 4:36 AM
> >
> > >
> >
> > > > >> To: users@nifi.apache.org
> >
> > >
> >
> > > > >> Subject: Re: NiFi fails on cluster nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Can you see if zookeeper node is up and running and can connect
> >
> > > > >> to
> >
> > >
> >
> > > > >> the nifi nodes
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Hello,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,748 INFO [main]
> >
> > >
> >
> > > > >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties
> >
> > > > >> from
> >
> > >
> >
> > > > >> /opt/nifi-1.7.1/./conf/nifi.properties
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded
> >
> > > > >> 125
> >
> > >
> >
> > > > >> properties
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:57:07,762 INFO [main]
> >
> > >
> >
> > > > >> org.apache.nifi.BootstrapListener Started Bootstrap Listener,
> >
> > >
> >
> > > > >> Listening for incoming requests on port
> >
> > >
> >
> > > > >> 43744
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi
> > > > >> Failure
> >
> > >
> >
> > > > >> to launch NiFi due to java.net.ConnectException: Connection
> > > > >> timed
> >
> > >
> >
> > > > >> out (Connection timed out)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> java.net.ConnectException: Connection timed out (Connection
> > > > >> timed
> >
> > >
> >
> > > > >> out)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at java.net.PlainSocketImpl.socketConnect(Native
> > > > >> Method)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.
> >
> > >
> >
> > > > >> ja
> >
> > >
> >
> > > > >> va:350)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlain
> > > > >> So
> >
> > > > >> ck
> >
> > >
> >
> > > > >> et
> >
> > >
> >
> > > > >> Impl.java:206)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.
> >
> > > > >> ja
> >
> > >
> >
> > > > >> va
> >
> > >
> >
> > > > >> :188)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at java.net.Socket.connect(Socket.java:589)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at java.net.Socket.connect(Socket.java:538)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
> >
> > >
> >
> > > > >> 100)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at
> >
> > >
> >
> > > > >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:
> > > > >> 83
> >
> > > > >> )
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> at org.apache.nifi.NiFi.main(NiFi.java:292)
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Thanks in advance,
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >>
> >
> > >
> >
> > > > >> Alexander
>
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Thanks again, Bryan. Just a quick follow-up question: does removing users.xml and authorizations.xml mean that we will need to re-create all users and groups that we had in the original standalone NiFi instance?
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Monday, October 22, 2018 12:48 PM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
Sorry I was confused when you said two 1 node clusters and I assumed they each had their own ZooKeeper.
You don't need to run ZK on both nodes, you can create a 2 node cluster using the embedded ZK on the first node.
This blog post shows how to setup a secure 2 node cluster:
https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
The only difference is that the authorizers.xml has changed slightly, so instead of:
<authorizer>
<identifier>file-provider</identifier>
<class>org.apache.nifi.authorization.FileAuthorizer</class>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Users File">./conf/users.xml</property>
<property name="Initial Admin Identity">CN=bbende, OU=ApacheNiFi</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=localhost, OU=NIFI</property> </authorizer>
You need to add the the users to the user-group-provider and then to the access-policy-provider...
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=bbende, OU=Apache NiFI</property>
<property name="Initial User Identity 2">CN=nifi-host-1, OU=NIFI</property>
<property name="Initial User Identity 2">CN=nifi-host-2, OU=NIFI</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group
Provider">composite-configurable-user-group-provider</property>
<property name="Authorizations
File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=bbende, OU=Apache NiFI</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=nifi-host-1, OU=NIFI</property>
<property name="Node Identity 1">CN=nifi-host-2, OU=NIFI</property>
</accessPolicyProvider>
Also, whenever you change any config in the authorizers.xml related to the file-based providers, then you will need to remove users.xml and authorizations.xml On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> Hi Bryan,
>
>
>
> At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
>
>
> Are you saying that we have to run ZooKeeper on both nodes? BTW, do we
> still need
>
>
>
> nifi.login.identity.provider.configuration.file=./conf/login-identity-
> providers.xml
>
>
>
> in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
>
>
> Thank you,
>
>
>
> Alexander
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Monday, October 22, 2018 11:55 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
>
>
> In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
>
>
> server.1=nifi-node1-hostname:2888:3888
>
> server.2=nifi-node2-hostname:2888:3888
>
> On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > I wonder if anyone has run into the same problem when trying to
>
> > configure composite authentication/authorization (LDAP and local
>
> > file)? When we use the “stand-alone” authorizers.xml file with the
>
> > addition of two extra properties
>
> >
>
> >
>
> >
>
> > <property name="Node Identity 1">…
>
> >
>
> > <property name="Node Identity 2">…
>
> >
>
> >
>
> >
>
> > and let ZooKeeper start on one on the nodes, we end up with two
>
> > one-node clusters, since apparently, the NiFi instances don’t talk
> > to
>
> > each other, but at least, they come alive…
>
> >
>
> >
>
> >
>
> > From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
>
> > Sent: Friday, October 19, 2018 11:18 AM
>
> > To: users@nifi.apache.org
>
> > Subject: RE: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
> >
>
> >
>
> >
>
> > 2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader
>
> > Context initialization failed
>
> >
>
> > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > Error creating bean with name
>
> > 'org.springframework.security.config.annotation.web.configuration.We
> > bS
>
> > ecurityConfiguration': Unsatisfied dependency expressed through
> > method
>
> > 'setFilterChainProxySecurityConfigurer' parameter 1; nested
> > exception
>
> > is org.springframework.beans.factory.BeanExpressionException:
>
> > Expression parsing failed; nested exception is
>
> > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > Error creating bean with name
>
> > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>
> > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > parameter 0; nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'jwtAuthenticationProvider' defined in class
>
> > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > reference to bean 'authorizer' while setting constructor argument;
>
> > nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'authorizer': FactoryBean threw exception on
>
> > object creation; nested exception is java.lang.NullPointerException:
>
> > Name is null
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > Po
>
> > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPos
> > tP
>
> > rocessor.java:667)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.InjectionMetadata.injec
> > t(
>
> > InjectionMetadata.java:88)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > Po
>
> > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostPro
> > ce
>
> > ssor.java:366)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > nF
>
> > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > nF
>
> > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > nF
>
> > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory$1.getO
> > bj
>
> > ect(AbstractBeanFactory.java:306)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultSingletonBeanRegist
> > ry
>
> > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.doGetB
> > ea
>
> > n(AbstractBeanFactory.java:302)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.getBea
> > n(
>
> > AbstractBeanFactory.java:197)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > .p
>
> > reInstantiateSingletons(DefaultListableBeanFactory.java:761)
>
> >
>
> > at
>
> > org.springframework.context.support.AbstractApplicationContext.finis
> > hB
>
> > eanFactoryInitialization(AbstractApplicationContext.java:867)
>
> >
>
> > at
>
> > org.springframework.context.support.AbstractApplicationContext.refre
> > sh
>
> > (AbstractApplicationContext.java:543)
>
> >
>
> > at
>
> > org.springframework.web.context.ContextLoader.configureAndRefreshWeb
> > Ap
>
> > plicationContext(ContextLoader.java:443)
>
> >
>
> > at
>
> > org.springframework.web.context.ContextLoader.initWebApplicationCont
> > ex
>
> > t(ContextLoader.java:325)
>
> >
>
> > at
>
> > org.springframework.web.context.ContextLoaderListener.contextInitial
> > iz
>
> > ed(ContextLoaderListener.java:107)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.ContextHandler.callContextInitializ
> > ed
>
> > (ContextHandler.java:876)
>
> >
>
> > at
>
> > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitializ
> > ed
>
> > (ServletContextHandler.java:532)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.ContextHandler.startContext(Context
> > Ha
>
> > ndler.java:839)
>
> >
>
> > at
>
> > org.eclipse.jetty.servlet.ServletContextHandler.startContext(Servlet
> > Co
>
> > ntextHandler.java:344)
>
> >
>
> > at
>
> > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:
>
> > 1480)
>
> >
>
> > at
>
> > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.ja
> > va
>
> > :1442)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandl
> > er
>
> > .java:799)
>
> >
>
> > at
>
> > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletConte
> > xt
>
> > Handler.java:261)
>
> >
>
> > at
>
> > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:54
> > 0)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > eC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL
> > if
>
> > eCycle.java:131)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe
> > rL
>
> > ifeCycle.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan
> > dl
>
> > er.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > eC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL
> > if
>
> > eCycle.java:131)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe
> > rL
>
> > ifeCycle.java:105)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan
> > dl
>
> > er.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.
>
> > java:290)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > eC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL
> > if
>
> > eCycle.java:131)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe
> > rL
>
> > ifeCycle.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan
> > dl
>
> > er.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > eC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerL
> > if
>
> > eCycle.java:131)
>
> >
>
> > at org.eclipse.jetty.server.Server.start(Server.java:452)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(Containe
> > rL
>
> > ifeCycle.java:105)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHan
> > dl
>
> > er.java:113)
>
> >
>
> > at org.eclipse.jetty.server.Server.doStart(Server.java:419)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLif
> > eC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838)
>
> >
>
> > at org.apache.nifi.NiFi.<init>(NiFi.java:157)
>
> >
>
> > at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> >
>
> > at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> >
>
> > Caused by: org.springframework.beans.factory.BeanExpressionException:
>
> > Expression parsing failed; nested exception is
>
> > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > Error creating bean with name
>
> > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>
> > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > parameter 0; nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'jwtAuthenticationProvider' defined in class
>
> > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > reference to bean 'authorizer' while setting constructor argument;
>
> > nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'authorizer': FactoryBean threw exception on
>
> > object creation; nested exception is java.lang.NullPointerException:
>
> > Name is null
>
> >
>
> > at
>
> > org.springframework.context.expression.StandardBeanExpressionResolver.
>
> > evaluate(StandardBeanExpressionResolver.java:164)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.evalua
> > te
>
> > BeanDefinitionString(AbstractBeanFactory.java:1448)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > .d
>
> > oResolveDependency(DefaultListableBeanFactory.java:1088)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > .r
>
> > esolveDependency(DefaultListableBeanFactory.java:1066)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > Po
>
> > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPos
> > tP
>
> > rocessor.java:659)
>
> >
>
> > ... 48 common frames omitted
>
> >
>
> > Caused by:
>
> > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > Error creating bean with name
>
> > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>
> > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > parameter 0; nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'jwtAuthenticationProvider' defined in class
>
> > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > reference to bean 'authorizer' while setting constructor argument;
>
> > nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'authorizer': FactoryBean threw exception on
>
> > object creation; nested exception is java.lang.NullPointerException:
>
> > Name is null
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > Po
>
> > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPos
> > tP
>
> > rocessor.java:667)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.InjectionMetadata.injec
> > t(
>
> > InjectionMetadata.java:88)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBean
> > Po
>
> > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostPro
> > ce
>
> > ssor.java:366)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > nF
>
> > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > nF
>
> > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBea
> > nF
>
> > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory$1.getO
> > bj
>
> > ect(AbstractBeanFactory.java:306)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultSingletonBeanRegist
> > ry
>
> > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.doGetB
> > ea
>
> > n(AbstractBeanFactory.java:302)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.getBea
> > n(
>
> > AbstractBeanFactory.java:202)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > .g
>
> > etBeansOfType(DefaultListableBeanFactory.java:519)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory
> > .g
>
> > etBeansOfType(DefaultListableBeanFactory.java:508)
>
> >
>
> > at
>
> > org.springframework.security.config.annotation.web.configuration.Aut
> > ow
>
> > iredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(Au
> > to
>
> > wiredWebSecurityConfigurersIgnoreParents.java:53)
>
> >
>
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
>
> >
>
> > at
>
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
> > .j
>
> > ava:62)
>
> >
>
> > at
>
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
> > ss
>
> > orImpl.java:43)
>
> >
>
> > at java.lang.reflect.Method.invoke(Method.java:498)
>
> >
>
> > at
>
> > org.springframework.expression.spel.support.ReflectiveMethodExecutor
> > .e
>
> > xecute(ReflectiveMethodExecutor.java:113)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.MethodReference.getValueInte
> > rn
>
> > al(MethodReference.java:129)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.MethodReference.access$000(M
> > et
>
> > hodReference.java:49)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.MethodReference$MethodValueR
> > ef
>
> > .getValue(MethodReference.java:347)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.CompoundExpression.getValueI
> > nt
>
> > ernal(CompoundExpression.java:88)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNo
> > de
>
> > Impl.java:120)
>
> >
>
> > at
>
> > org.springframework.expression.spel.standard.SpelExpression.getValue
> > (S
>
> > pelExpression.java:262)
>
> >
>
> > at
>
> > org.springframework.context.expression.StandardBeanExpressionResolver.
>
> > evaluate(StandardBeanExpressionResolver.java:161)
>
> >
>
> > ... 52 common frames omitted
>
> >
>
> >
>
> >
>
> > I tried to Google for possible clues, but so far, there hasn’t been
>
> > any luck…
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Monday, October 15, 2018 10:27 AM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
>
> >
>
> >
>
> >
>
> > In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
>
> >
>
> >
>
> >
>
> > Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
>
> >
>
> > On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > Sent: Monday, October 15, 2018 9:43 AM
>
> >
>
> > > To: users@nifi.apache.org
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
>
> >
>
> > >
>
> >
>
> > > On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> >
>
> > > >
>
> >
>
> > > > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
>
> >
>
> > > >
>
> >
>
> > > > Mike
>
> >
>
> > > >
>
> >
>
> > > > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Hi Mike and Bryan,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol
>
> > > >> Request-1]
>
> >
>
> > > >> o.a.nifi.security.util.CertificateUtils The incoming request
> > > >> did
>
> >
>
> > > >> not contain client certificates and thus the DN cannot be extracted.
>
> >
>
> > > >> Check that the other endpoint is providing a complete client
>
> >
>
> > > >> certificate chain
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-15 09:19:48,425 INFO [main]
>
> >
>
> > > >> o.a.nifi.controller.StandardFlowService Connecting Node:
>
> >
>
> > > >> 0.0.0.0:8008
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol
>
> > > >> Request-2]
>
> >
>
> > > >> o.a.nifi.security.util.CertificateUtils The incoming request
> > > >> did
>
> >
>
> > > >> not contain client certificates and thus the DN cannot be extracted.
>
> >
>
> > > >> Check that the other endpoint is providing a complete client
>
> >
>
> > > >> certificate chain
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-15 09:19:48,456 WARN [main]
>
> >
>
> > > >> o.a.nifi.controller.StandardFlowService Failed to connect to
>
> >
>
> > > >> cluster due to: org.apache.nifi.cluster.protocol.ProtocolException:
>
> >
>
> > > >> Failed marshalling 'CONNECTION_REQUEST' protocol message due to:
>
> >
>
> > > >> javax.net.ssl.SSLHandshakeException: Received fatal alert:
>
> >
>
> > > >> bad_certificate
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Thank you.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: Mike Thomsen <mi...@gmail.com>
>
> >
>
> > > >> Sent: Monday, October 15, 2018 9:02 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> http://nifi.apache.org/docs/nifi-docs/html/administration-guide
> > > >> .h
>
> > > >> tm
>
> >
>
> > > >> l
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> See the properties that start with "nifi.zookeeper."
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Mike,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Thanks again.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: Mike Thomsen <mi...@gmail.com>
>
> >
>
> > > >> Sent: Friday, October 12, 2018 10:17 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 08:21:35,838 WARN [main]
>
> >
>
> > > >> o.a.nifi.controller.StandardFlowService There is currently no
>
> >
>
> > > >> Cluster Coordinator. This often happens upon restart of NiFi
> > > >> when
>
> >
>
> > > >> running an embedded ZooKeeper. Will register this node to
> > > >> become
>
> >
>
> > > >> the active Cluster Coordinator and will attempt to connect to
>
> >
>
> > > >> cluster again
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 08:21:35,838 INFO [main]
>
> >
>
> > > >> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> >
>
> > > >> CuratorLeaderElectionManager[stopped=false] Attempted to
> > > >> register
>
> >
>
> > > >> Leader Election for role 'Cluster Coordinator' but this role is
>
> >
>
> > > >> already registered
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
>
> >
>
> > > >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
>
> >
>
> > > >> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> >
>
> > > >> org.apache.nifi.controller.leader.election.CuratorLeaderElectio
> > > >> nM
>
> > > >> an
>
> >
>
> > > >> ag er$ElectionListener@17900f5b Connection State changed to
>
> >
>
> > > >> SUSPENDED
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: Mike Thomsen <mi...@gmail.com>
>
> >
>
> > > >> Sent: Friday, October 12, 2018 8:33 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Alexander,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> I am pretty sure your problem is here:
>
> >
>
> > > >> nifi.state.management.embedded.zookeeper.start=true
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> What you need to try is these steps:
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 1. Set up an external ZooKeeper instance (or set up 3 in a
>
> > > >> quorum;
>
> >
>
> > > >> must be odd numbers)
>
> >
>
> > > >>
>
> >
>
> > > >> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>
> >
>
> > > >>
>
> >
>
> > > >> 3. Restart all of them.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> See if that works.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Mike
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
>
> >
>
> > > >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry
> > > >> gave
>
> >
>
> > > >> up
>
> >
>
> > > >>
>
> >
>
> > > >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>
> >
>
> > > >> KeeperErrorCode = ConnectionLoss
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:
>
> > > >> 99
>
> >
>
> > > >> )
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBac
> > > >> kg
>
> > > >> ro
>
> >
>
> > > >> un
>
> >
>
> > > >> dRetry(CuratorFrameworkImpl.java:728)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.performB
> > > >> ac
>
> > > >> kg
>
> >
>
> > > >> ro
>
> >
>
> > > >> undOperation(CuratorFrameworkImpl.java:857)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgrou
> > > >> nd
>
> > > >> Op
>
> >
>
> > > >> er
>
> >
>
> > > >> ationsLoop(CuratorFrameworkImpl.java:809)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$3
> > > >> 00
>
> > > >> (C
>
> >
>
> > > >> ur
>
> >
>
> > > >> atorFrameworkImpl.java:64)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(C
> > > >> ur
>
> > > >> at
>
> >
>
> > > >> or
>
> >
>
> > > >> FrameworkImpl.java:267)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> > > >> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>
> >
>
> > > >> access$201(ScheduledThreadPoolExecutor.java:180)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>
> >
>
> > > >> run(ScheduledThreadPoolExecutor.java:293)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
>
> >
>
> > > >> java:1149)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx
> > > >> ec
>
> > > >> ut
>
> >
>
> > > >> or
>
> >
>
> > > >> .java:624)
>
> >
>
> > > >>
>
> >
>
> > > >> at java.lang.Thread.run(Thread.java:748)
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Is there anything else we should check?
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: Nathan Gough <th...@gmail.com>
>
> >
>
> > > >> Sent: Thursday, October 11, 2018 9:12 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: ashmeet kandhari <as...@gmail.com>
>
> >
>
> > > >> Reply-To: <us...@nifi.apache.org>
>
> >
>
> > > >> Date: Thursday, October 11, 2018 at 9:09 AM
>
> >
>
> > > >> To: <us...@nifi.apache.org>
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Hi Alexander,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: ashmeet kandhari <as...@gmail.com>
>
> >
>
> > > >> Sent: Thursday, October 11, 2018 4:36 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Can you see if zookeeper node is up and running and can connect
>
> > > >> to
>
> >
>
> > > >> the nifi nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Hello,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,748 INFO [main]
>
> >
>
> > > >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties
>
> > > >> from
>
> >
>
> > > >> /opt/nifi-1.7.1/./conf/nifi.properties
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded
>
> > > >> 125
>
> >
>
> > > >> properties
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,762 INFO [main]
>
> >
>
> > > >> org.apache.nifi.BootstrapListener Started Bootstrap Listener,
>
> >
>
> > > >> Listening for incoming requests on port
>
> >
>
> > > >> 43744
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi
> > > >> Failure
>
> >
>
> > > >> to launch NiFi due to java.net.ConnectException: Connection
> > > >> timed
>
> >
>
> > > >> out (Connection timed out)
>
> >
>
> > > >>
>
> >
>
> > > >> java.net.ConnectException: Connection timed out (Connection
> > > >> timed
>
> >
>
> > > >> out)
>
> >
>
> > > >>
>
> >
>
> > > >> at java.net.PlainSocketImpl.socketConnect(Native
> > > >> Method)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.
>
> >
>
> > > >> ja
>
> >
>
> > > >> va:350)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlain
> > > >> So
>
> > > >> ck
>
> >
>
> > > >> et
>
> >
>
> > > >> Impl.java:206)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.
>
> > > >> ja
>
> >
>
> > > >> va
>
> >
>
> > > >> :188)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> >
>
> > > >>
>
> >
>
> > > >> at java.net.Socket.connect(Socket.java:589)
>
> >
>
> > > >>
>
> >
>
> > > >> at java.net.Socket.connect(Socket.java:538)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
>
> >
>
> > > >> 100)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:
> > > >> 83
>
> > > >> )
>
> >
>
> > > >>
>
> >
>
> > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> >
>
> > > >>
>
> >
>
> > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> >
>
> > > >>
>
> >
>
> > > >> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Thanks in advance,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Alexander
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
Sorry I was confused when you said two 1 node clusters and I assumed
they each had their own ZooKeeper.
You don't need to run ZK on both nodes, you can create a 2 node
cluster using the embedded ZK on the first node.
This blog post shows how to setup a secure 2 node cluster:
https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
The only difference is that the authorizers.xml has changed slightly,
so instead of:
<authorizer>
<identifier>file-provider</identifier>
<class>org.apache.nifi.authorization.FileAuthorizer</class>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Users File">./conf/users.xml</property>
<property name="Initial Admin Identity">CN=bbende, OU=ApacheNiFi</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=localhost, OU=NIFI</property>
</authorizer>
You need to add the the users to the user-group-provider and then to
the access-policy-provider...
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=bbende, OU=Apache
NiFI</property>
<property name="Initial User Identity 2">CN=nifi-host-1,
OU=NIFI</property>
<property name="Initial User Identity 2">CN=nifi-host-2,
OU=NIFI</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group
Provider">composite-configurable-user-group-provider</property>
<property name="Authorizations
File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=bbende, OU=Apache
NiFI</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=nifi-host-1, OU=NIFI</property>
<property name="Node Identity 1">CN=nifi-host-2, OU=NIFI</property>
</accessPolicyProvider>
Also, whenever you change any config in the authorizers.xml related to
the file-based providers, then you will need to remove users.xml and
authorizations.xml
On Mon, Oct 22, 2018 at 12:20 PM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> Hi Bryan,
>
>
>
> At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
>
>
>
> Are you saying that we have to run ZooKeeper on both nodes? BTW, do we still need
>
>
>
> nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
>
>
>
> in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
>
>
>
> Thank you,
>
>
>
> Alexander
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Monday, October 22, 2018 11:55 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
>
>
>
> In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
>
>
>
> server.1=nifi-node1-hostname:2888:3888
>
> server.2=nifi-node2-hostname:2888:3888
>
> On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > I wonder if anyone has run into the same problem when trying to
>
> > configure composite authentication/authorization (LDAP and local
>
> > file)? When we use the “stand-alone” authorizers.xml file with the
>
> > addition of two extra properties
>
> >
>
> >
>
> >
>
> > <property name="Node Identity 1">…
>
> >
>
> > <property name="Node Identity 2">…
>
> >
>
> >
>
> >
>
> > and let ZooKeeper start on one on the nodes, we end up with two
>
> > one-node clusters, since apparently, the NiFi instances don’t talk to
>
> > each other, but at least, they come alive…
>
> >
>
> >
>
> >
>
> > From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
>
> > Sent: Friday, October 19, 2018 11:18 AM
>
> > To: users@nifi.apache.org
>
> > Subject: RE: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
> >
>
> >
>
> >
>
> > 2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader
>
> > Context initialization failed
>
> >
>
> > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > Error creating bean with name
>
> > 'org.springframework.security.config.annotation.web.configuration.WebS
>
> > ecurityConfiguration': Unsatisfied dependency expressed through method
>
> > 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception
>
> > is org.springframework.beans.factory.BeanExpressionException:
>
> > Expression parsing failed; nested exception is
>
> > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > Error creating bean with name
>
> > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>
> > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > parameter 0; nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'jwtAuthenticationProvider' defined in class
>
> > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > reference to bean 'authorizer' while setting constructor argument;
>
> > nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'authorizer': FactoryBean threw exception on
>
> > object creation; nested exception is java.lang.NullPointerException:
>
> > Name is null
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
>
> > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostP
>
> > rocessor.java:667)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.InjectionMetadata.inject(
>
> > InjectionMetadata.java:88)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
>
> > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProce
>
> > ssor.java:366)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
>
> > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
>
> > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
>
> > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory$1.getObj
>
> > ect(AbstractBeanFactory.java:306)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry
>
> > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBea
>
> > n(AbstractBeanFactory.java:302)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(
>
> > AbstractBeanFactory.java:197)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory.p
>
> > reInstantiateSingletons(DefaultListableBeanFactory.java:761)
>
> >
>
> > at
>
> > org.springframework.context.support.AbstractApplicationContext.finishB
>
> > eanFactoryInitialization(AbstractApplicationContext.java:867)
>
> >
>
> > at
>
> > org.springframework.context.support.AbstractApplicationContext.refresh
>
> > (AbstractApplicationContext.java:543)
>
> >
>
> > at
>
> > org.springframework.web.context.ContextLoader.configureAndRefreshWebAp
>
> > plicationContext(ContextLoader.java:443)
>
> >
>
> > at
>
> > org.springframework.web.context.ContextLoader.initWebApplicationContex
>
> > t(ContextLoader.java:325)
>
> >
>
> > at
>
> > org.springframework.web.context.ContextLoaderListener.contextInitializ
>
> > ed(ContextLoaderListener.java:107)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized
>
> > (ContextHandler.java:876)
>
> >
>
> > at
>
> > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized
>
> > (ServletContextHandler.java:532)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHa
>
> > ndler.java:839)
>
> >
>
> > at
>
> > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletCo
>
> > ntextHandler.java:344)
>
> >
>
> > at
>
> > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:
>
> > 1480)
>
> >
>
> > at
>
> > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java
>
> > :1442)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler
>
> > .java:799)
>
> >
>
> > at
>
> > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContext
>
> > Handler.java:261)
>
> >
>
> > at
>
> > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLif
>
> > eCycle.java:131)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerL
>
> > ifeCycle.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandl
>
> > er.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLif
>
> > eCycle.java:131)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerL
>
> > ifeCycle.java:105)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandl
>
> > er.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.
>
> > java:290)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLif
>
> > eCycle.java:131)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerL
>
> > ifeCycle.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandl
>
> > er.java:113)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLif
>
> > eCycle.java:131)
>
> >
>
> > at org.eclipse.jetty.server.Server.start(Server.java:452)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerL
>
> > ifeCycle.java:105)
>
> >
>
> > at
>
> > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandl
>
> > er.java:113)
>
> >
>
> > at org.eclipse.jetty.server.Server.doStart(Server.java:419)
>
> >
>
> > at
>
> > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
>
> > ycle.java:68)
>
> >
>
> > at
>
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838)
>
> >
>
> > at org.apache.nifi.NiFi.<init>(NiFi.java:157)
>
> >
>
> > at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> >
>
> > at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> >
>
> > Caused by: org.springframework.beans.factory.BeanExpressionException:
>
> > Expression parsing failed; nested exception is
>
> > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > Error creating bean with name
>
> > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>
> > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > parameter 0; nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'jwtAuthenticationProvider' defined in class
>
> > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > reference to bean 'authorizer' while setting constructor argument;
>
> > nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'authorizer': FactoryBean threw exception on
>
> > object creation; nested exception is java.lang.NullPointerException:
>
> > Name is null
>
> >
>
> > at
>
> > org.springframework.context.expression.StandardBeanExpressionResolver.
>
> > evaluate(StandardBeanExpressionResolver.java:164)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.evaluate
>
> > BeanDefinitionString(AbstractBeanFactory.java:1448)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory.d
>
> > oResolveDependency(DefaultListableBeanFactory.java:1088)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory.r
>
> > esolveDependency(DefaultListableBeanFactory.java:1066)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
>
> > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostP
>
> > rocessor.java:659)
>
> >
>
> > ... 48 common frames omitted
>
> >
>
> > Caused by:
>
> > org.springframework.beans.factory.UnsatisfiedDependencyException:
>
> > Error creating bean with name
>
> > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
>
> > dependency expressed through method 'setJwtAuthenticationProvider'
>
> > parameter 0; nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'jwtAuthenticationProvider' defined in class
>
> > path resource [nifi-web-security-context.xml]: Cannot resolve
>
> > reference to bean 'authorizer' while setting constructor argument;
>
> > nested exception is
>
> > org.springframework.beans.factory.BeanCreationException: Error
>
> > creating bean with name 'authorizer': FactoryBean threw exception on
>
> > object creation; nested exception is java.lang.NullPointerException:
>
> > Name is null
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
>
> > stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostP
>
> > rocessor.java:667)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.InjectionMetadata.inject(
>
> > InjectionMetadata.java:88)
>
> >
>
> > at
>
> > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
>
> > stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProce
>
> > ssor.java:366)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
>
> > actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
>
> > actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
>
> > actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory$1.getObj
>
> > ect(AbstractBeanFactory.java:306)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry
>
> > .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBea
>
> > n(AbstractBeanFactory.java:302)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(
>
> > AbstractBeanFactory.java:202)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory.g
>
> > etBeansOfType(DefaultListableBeanFactory.java:519)
>
> >
>
> > at
>
> > org.springframework.beans.factory.support.DefaultListableBeanFactory.g
>
> > etBeansOfType(DefaultListableBeanFactory.java:508)
>
> >
>
> > at
>
> > org.springframework.security.config.annotation.web.configuration.Autow
>
> > iredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(Auto
>
> > wiredWebSecurityConfigurersIgnoreParents.java:53)
>
> >
>
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> >
>
> > at
>
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
>
> > ava:62)
>
> >
>
> > at
>
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
>
> > orImpl.java:43)
>
> >
>
> > at java.lang.reflect.Method.invoke(Method.java:498)
>
> >
>
> > at
>
> > org.springframework.expression.spel.support.ReflectiveMethodExecutor.e
>
> > xecute(ReflectiveMethodExecutor.java:113)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.MethodReference.getValueIntern
>
> > al(MethodReference.java:129)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.MethodReference.access$000(Met
>
> > hodReference.java:49)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.MethodReference$MethodValueRef
>
> > .getValue(MethodReference.java:347)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.CompoundExpression.getValueInt
>
> > ernal(CompoundExpression.java:88)
>
> >
>
> > at
>
> > org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNode
>
> > Impl.java:120)
>
> >
>
> > at
>
> > org.springframework.expression.spel.standard.SpelExpression.getValue(S
>
> > pelExpression.java:262)
>
> >
>
> > at
>
> > org.springframework.context.expression.StandardBeanExpressionResolver.
>
> > evaluate(StandardBeanExpressionResolver.java:161)
>
> >
>
> > ... 52 common frames omitted
>
> >
>
> >
>
> >
>
> > I tried to Google for possible clues, but so far, there hasn’t been
>
> > any luck…
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Monday, October 15, 2018 10:27 AM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> >
>
> >
>
> > I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
>
> >
>
> >
>
> >
>
> > In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
>
> >
>
> >
>
> >
>
> > Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
>
> >
>
> > On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > >
>
> >
>
> > > Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: Bryan Bende <bb...@gmail.com>
>
> >
>
> > > Sent: Monday, October 15, 2018 9:43 AM
>
> >
>
> > > To: users@nifi.apache.org
>
> >
>
> > > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > >
>
> >
>
> > > This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
>
> >
>
> > >
>
> >
>
> > > On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> >
>
> > > >
>
> >
>
> > > > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
>
> >
>
> > > >
>
> >
>
> > > > Mike
>
> >
>
> > > >
>
> >
>
> > > > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Hi Mike and Bryan,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol
>
> > > >> Request-1]
>
> >
>
> > > >> o.a.nifi.security.util.CertificateUtils The incoming request did
>
> >
>
> > > >> not contain client certificates and thus the DN cannot be extracted.
>
> >
>
> > > >> Check that the other endpoint is providing a complete client
>
> >
>
> > > >> certificate chain
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-15 09:19:48,425 INFO [main]
>
> >
>
> > > >> o.a.nifi.controller.StandardFlowService Connecting Node:
>
> >
>
> > > >> 0.0.0.0:8008
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol
>
> > > >> Request-2]
>
> >
>
> > > >> o.a.nifi.security.util.CertificateUtils The incoming request did
>
> >
>
> > > >> not contain client certificates and thus the DN cannot be extracted.
>
> >
>
> > > >> Check that the other endpoint is providing a complete client
>
> >
>
> > > >> certificate chain
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-15 09:19:48,456 WARN [main]
>
> >
>
> > > >> o.a.nifi.controller.StandardFlowService Failed to connect to
>
> >
>
> > > >> cluster due to: org.apache.nifi.cluster.protocol.ProtocolException:
>
> >
>
> > > >> Failed marshalling 'CONNECTION_REQUEST' protocol message due to:
>
> >
>
> > > >> javax.net.ssl.SSLHandshakeException: Received fatal alert:
>
> >
>
> > > >> bad_certificate
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Thank you.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: Mike Thomsen <mi...@gmail.com>
>
> >
>
> > > >> Sent: Monday, October 15, 2018 9:02 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> http://nifi.apache.org/docs/nifi-docs/html/administration-guide.h
>
> > > >> tm
>
> >
>
> > > >> l
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> See the properties that start with "nifi.zookeeper."
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Mike,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Thanks again.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: Mike Thomsen <mi...@gmail.com>
>
> >
>
> > > >> Sent: Friday, October 12, 2018 10:17 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 08:21:35,838 WARN [main]
>
> >
>
> > > >> o.a.nifi.controller.StandardFlowService There is currently no
>
> >
>
> > > >> Cluster Coordinator. This often happens upon restart of NiFi when
>
> >
>
> > > >> running an embedded ZooKeeper. Will register this node to become
>
> >
>
> > > >> the active Cluster Coordinator and will attempt to connect to
>
> >
>
> > > >> cluster again
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 08:21:35,838 INFO [main]
>
> >
>
> > > >> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> >
>
> > > >> CuratorLeaderElectionManager[stopped=false] Attempted to register
>
> >
>
> > > >> Leader Election for role 'Cluster Coordinator' but this role is
>
> >
>
> > > >> already registered
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
>
> >
>
> > > >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
>
> >
>
> > > >> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> >
>
> > > >> org.apache.nifi.controller.leader.election.CuratorLeaderElectionM
>
> > > >> an
>
> >
>
> > > >> ag er$ElectionListener@17900f5b Connection State changed to
>
> >
>
> > > >> SUSPENDED
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: Mike Thomsen <mi...@gmail.com>
>
> >
>
> > > >> Sent: Friday, October 12, 2018 8:33 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Alexander,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> I am pretty sure your problem is here:
>
> >
>
> > > >> nifi.state.management.embedded.zookeeper.start=true
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> What you need to try is these steps:
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 1. Set up an external ZooKeeper instance (or set up 3 in a
>
> > > >> quorum;
>
> >
>
> > > >> must be odd numbers)
>
> >
>
> > > >>
>
> >
>
> > > >> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>
> >
>
> > > >>
>
> >
>
> > > >> 3. Restart all of them.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> See if that works.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Mike
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
>
> >
>
> > > >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave
>
> >
>
> > > >> up
>
> >
>
> > > >>
>
> >
>
> > > >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>
> >
>
> > > >> KeeperErrorCode = ConnectionLoss
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:
>
> > > >> 99
>
> >
>
> > > >> )
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackg
>
> > > >> ro
>
> >
>
> > > >> un
>
> >
>
> > > >> dRetry(CuratorFrameworkImpl.java:728)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBac
>
> > > >> kg
>
> >
>
> > > >> ro
>
> >
>
> > > >> undOperation(CuratorFrameworkImpl.java:857)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.background
>
> > > >> Op
>
> >
>
> > > >> er
>
> >
>
> > > >> ationsLoop(CuratorFrameworkImpl.java:809)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300
>
> > > >> (C
>
> >
>
> > > >> ur
>
> >
>
> > > >> atorFrameworkImpl.java:64)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(Cur
>
> > > >> at
>
> >
>
> > > >> or
>
> >
>
> > > >> FrameworkImpl.java:267)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> > > >> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>
> >
>
> > > >> access$201(ScheduledThreadPoolExecutor.java:180)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>
> >
>
> > > >> run(ScheduledThreadPoolExecutor.java:293)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
>
> >
>
> > > >> java:1149)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec
>
> > > >> ut
>
> >
>
> > > >> or
>
> >
>
> > > >> .java:624)
>
> >
>
> > > >>
>
> >
>
> > > >> at java.lang.Thread.run(Thread.java:748)
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Is there anything else we should check?
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: Nathan Gough <th...@gmail.com>
>
> >
>
> > > >> Sent: Thursday, October 11, 2018 9:12 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: ashmeet kandhari <as...@gmail.com>
>
> >
>
> > > >> Reply-To: <us...@nifi.apache.org>
>
> >
>
> > > >> Date: Thursday, October 11, 2018 at 9:09 AM
>
> >
>
> > > >> To: <us...@nifi.apache.org>
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Hi Alexander,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> From: ashmeet kandhari <as...@gmail.com>
>
> >
>
> > > >> Sent: Thursday, October 11, 2018 4:36 AM
>
> >
>
> > > >> To: users@nifi.apache.org
>
> >
>
> > > >> Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Can you see if zookeeper node is up and running and can connect
>
> > > >> to
>
> >
>
> > > >> the nifi nodes
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > > >>
>
> >
>
> > > >> Hello,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,748 INFO [main]
>
> >
>
> > > >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties
>
> > > >> from
>
> >
>
> > > >> /opt/nifi-1.7.1/./conf/nifi.properties
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded
>
> > > >> 125
>
> >
>
> > > >> properties
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:57:07,762 INFO [main]
>
> >
>
> > > >> org.apache.nifi.BootstrapListener Started Bootstrap Listener,
>
> >
>
> > > >> Listening for incoming requests on port
>
> >
>
> > > >> 43744
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure
>
> >
>
> > > >> to launch NiFi due to java.net.ConnectException: Connection timed
>
> >
>
> > > >> out (Connection timed out)
>
> >
>
> > > >>
>
> >
>
> > > >> java.net.ConnectException: Connection timed out (Connection timed
>
> >
>
> > > >> out)
>
> >
>
> > > >>
>
> >
>
> > > >> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.
>
> >
>
> > > >> ja
>
> >
>
> > > >> va:350)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSo
>
> > > >> ck
>
> >
>
> > > >> et
>
> >
>
> > > >> Impl.java:206)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.
>
> > > >> ja
>
> >
>
> > > >> va
>
> >
>
> > > >> :188)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> >
>
> > > >>
>
> >
>
> > > >> at java.net.Socket.connect(Socket.java:589)
>
> >
>
> > > >>
>
> >
>
> > > >> at java.net.Socket.connect(Socket.java:538)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
>
> >
>
> > > >> 100)
>
> >
>
> > > >>
>
> >
>
> > > >> at
>
> >
>
> > > >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83
>
> > > >> )
>
> >
>
> > > >>
>
> >
>
> > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> >
>
> > > >>
>
> >
>
> > > >> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> >
>
> > > >>
>
> >
>
> > > >> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
>
> >
>
> > > >>
>
> >
>
> > > >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Thanks in advance,
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> Alexander
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Hi Bryan,
At this point, we don't want to run ZooKeeper on both nodes (as far as I understand, it prefers an odd number of members in the ensemble). Actually, the ZooKeeper running on one of them, sees both NiFi instances, but they don't talk to each other. When we try to make them do so by using a different authorizers.xml file, which is very much just a customized version of the “composite” example from the NiFi Admin Guide, then none of the nodes is able to start at all, throwing the error I mentioned in my previous post.
Are you saying that we have to run ZooKeeper on both nodes? BTW, do we still need
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
in the nifi.properties file when we use that new authorizers.xml? I’m asking since we have the same LDAP authentication/authorization settings in the latter.
Thank you,
Alexander
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Monday, October 22, 2018 11:55 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
If you are getting separate clusters then each node is likely only using it's own ZooKeeper and therefore doesn't know about the other node.
In nifi.properties the ZK connect string would need to be something like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in zoo.properties you would need entries for both ZooKeepers:
server.1=nifi-node1-hostname:2888:3888
server.2=nifi-node2-hostname:2888:3888
On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> I wonder if anyone has run into the same problem when trying to
> configure composite authentication/authorization (LDAP and local
> file)? When we use the “stand-alone” authorizers.xml file with the
> addition of two extra properties
>
>
>
> <property name="Node Identity 1">…
>
> <property name="Node Identity 2">…
>
>
>
> and let ZooKeeper start on one on the nodes, we end up with two
> one-node clusters, since apparently, the NiFi instances don’t talk to
> each other, but at least, they come alive…
>
>
>
> From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>>
> Sent: Friday, October 19, 2018 11:18 AM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: RE: NiFi fails on cluster nodes
>
>
>
> We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
>
>
> 2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader
> Context initialization failed
>
> org.springframework.beans.factory.UnsatisfiedDependencyException:
> Error creating bean with name
> 'org.springframework.security.config.annotation.web.configuration.WebS
> ecurityConfiguration': Unsatisfied dependency expressed through method
> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception
> is org.springframework.beans.factory.BeanExpressionException:
> Expression parsing failed; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException:
> Error creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> dependency expressed through method 'setJwtAuthenticationProvider'
> parameter 0; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'jwtAuthenticationProvider' defined in class
> path resource [nifi-web-security-context.xml]: Cannot resolve
> reference to bean 'authorizer' while setting constructor argument;
> nested exception is
> org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'authorizer': FactoryBean threw exception on
> object creation; nested exception is java.lang.NullPointerException:
> Name is null
>
> at
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
> stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostP
> rocessor.java:667)
>
> at
> org.springframework.beans.factory.annotation.InjectionMetadata.inject(
> InjectionMetadata.java:88)
>
> at
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
> stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProce
> ssor.java:366)
>
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
> actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
> actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
> actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObj
> ect(AbstractBeanFactory.java:306)
>
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry
> .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBea
> n(AbstractBeanFactory.java:302)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(
> AbstractBeanFactory.java:197)
>
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.p
> reInstantiateSingletons(DefaultListableBeanFactory.java:761)
>
> at
> org.springframework.context.support.AbstractApplicationContext.finishB
> eanFactoryInitialization(AbstractApplicationContext.java:867)
>
> at
> org.springframework.context.support.AbstractApplicationContext.refresh
> (AbstractApplicationContext.java:543)
>
> at
> org.springframework.web.context.ContextLoader.configureAndRefreshWebAp
> plicationContext(ContextLoader.java:443)
>
> at
> org.springframework.web.context.ContextLoader.initWebApplicationContex
> t(ContextLoader.java:325)
>
> at
> org.springframework.web.context.ContextLoaderListener.contextInitializ
> ed(ContextLoaderListener.java:107)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized
> (ContextHandler.java:876)
>
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized
> (ServletContextHandler.java:532)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHa
> ndler.java:839)
>
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletCo
> ntextHandler.java:344)
>
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:
> 1480)
>
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java
> :1442)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler
> .java:799)
>
> at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContext
> Handler.java:261)
>
> at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
> ycle.java:68)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLif
> eCycle.java:131)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerL
> ifeCycle.java:113)
>
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandl
> er.java:113)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
> ycle.java:68)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLif
> eCycle.java:131)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerL
> ifeCycle.java:105)
>
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandl
> er.java:113)
>
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.
> java:290)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
> ycle.java:68)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLif
> eCycle.java:131)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerL
> ifeCycle.java:113)
>
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandl
> er.java:113)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
> ycle.java:68)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLif
> eCycle.java:131)
>
> at org.eclipse.jetty.server.Server.start(Server.java:452)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerL
> ifeCycle.java:105)
>
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandl
> er.java:113)
>
> at org.eclipse.jetty.server.Server.doStart(Server.java:419)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
> ycle.java:68)
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:157)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> Caused by: org.springframework.beans.factory.BeanExpressionException:
> Expression parsing failed; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException:
> Error creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> dependency expressed through method 'setJwtAuthenticationProvider'
> parameter 0; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'jwtAuthenticationProvider' defined in class
> path resource [nifi-web-security-context.xml]: Cannot resolve
> reference to bean 'authorizer' while setting constructor argument;
> nested exception is
> org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'authorizer': FactoryBean threw exception on
> object creation; nested exception is java.lang.NullPointerException:
> Name is null
>
> at
> org.springframework.context.expression.StandardBeanExpressionResolver.
> evaluate(StandardBeanExpressionResolver.java:164)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.evaluate
> BeanDefinitionString(AbstractBeanFactory.java:1448)
>
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.d
> oResolveDependency(DefaultListableBeanFactory.java:1088)
>
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.r
> esolveDependency(DefaultListableBeanFactory.java:1066)
>
> at
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
> stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostP
> rocessor.java:659)
>
> ... 48 common frames omitted
>
> Caused by:
> org.springframework.beans.factory.UnsatisfiedDependencyException:
> Error creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> dependency expressed through method 'setJwtAuthenticationProvider'
> parameter 0; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'jwtAuthenticationProvider' defined in class
> path resource [nifi-web-security-context.xml]: Cannot resolve
> reference to bean 'authorizer' while setting constructor argument;
> nested exception is
> org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'authorizer': FactoryBean threw exception on
> object creation; nested exception is java.lang.NullPointerException:
> Name is null
>
> at
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
> stProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostP
> rocessor.java:667)
>
> at
> org.springframework.beans.factory.annotation.InjectionMetadata.inject(
> InjectionMetadata.java:88)
>
> at
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPo
> stProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProce
> ssor.java:366)
>
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
> actory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
> actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF
> actory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObj
> ect(AbstractBeanFactory.java:306)
>
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry
> .getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBea
> n(AbstractBeanFactory.java:302)
>
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(
> AbstractBeanFactory.java:202)
>
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.g
> etBeansOfType(DefaultListableBeanFactory.java:519)
>
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.g
> etBeansOfType(DefaultListableBeanFactory.java:508)
>
> at
> org.springframework.security.config.annotation.web.configuration.Autow
> iredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(Auto
> wiredWebSecurityConfigurersIgnoreParents.java:53)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
> org.springframework.expression.spel.support.ReflectiveMethodExecutor.e
> xecute(ReflectiveMethodExecutor.java:113)
>
> at
> org.springframework.expression.spel.ast.MethodReference.getValueIntern
> al(MethodReference.java:129)
>
> at
> org.springframework.expression.spel.ast.MethodReference.access$000(Met
> hodReference.java:49)
>
> at
> org.springframework.expression.spel.ast.MethodReference$MethodValueRef
> .getValue(MethodReference.java:347)
>
> at
> org.springframework.expression.spel.ast.CompoundExpression.getValueInt
> ernal(CompoundExpression.java:88)
>
> at
> org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNode
> Impl.java:120)
>
> at
> org.springframework.expression.spel.standard.SpelExpression.getValue(S
> pelExpression.java:262)
>
> at
> org.springframework.context.expression.StandardBeanExpressionResolver.
> evaluate(StandardBeanExpressionResolver.java:161)
>
> ... 52 common frames omitted
>
>
>
> I tried to Google for possible clues, but so far, there hasn’t been
> any luck…
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Monday, October 15, 2018 10:27 AM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
>
>
>
> In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
>
>
>
> Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
>
> On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> >
>
> > Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>>
>
> > Sent: Monday, October 15, 2018 9:43 AM
>
> > To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
>
> >
>
> > On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com>> wrote:
>
> > >
>
> > > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
>
> > >
>
> > > Mike
>
> > >
>
> > > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >>
>
> > >> Hi Mike and Bryan,
>
> > >>
>
> > >>
>
> > >>
>
> > >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
>
> > >>
>
> > >>
>
> > >>
>
> > >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol
> > >> Request-1]
>
> > >> o.a.nifi.security.util.CertificateUtils The incoming request did
>
> > >> not contain client certificates and thus the DN cannot be extracted.
>
> > >> Check that the other endpoint is providing a complete client
>
> > >> certificate chain
>
> > >>
>
> > >> 2018-10-15 09:19:48,425 INFO [main]
>
> > >> o.a.nifi.controller.StandardFlowService Connecting Node:
>
> > >> 0.0.0.0:8008
>
> > >>
>
> > >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol
> > >> Request-2]
>
> > >> o.a.nifi.security.util.CertificateUtils The incoming request did
>
> > >> not contain client certificates and thus the DN cannot be extracted.
>
> > >> Check that the other endpoint is providing a complete client
>
> > >> certificate chain
>
> > >>
>
> > >> 2018-10-15 09:19:48,456 WARN [main]
>
> > >> o.a.nifi.controller.StandardFlowService Failed to connect to
>
> > >> cluster due to: org.apache.nifi.cluster.protocol.ProtocolException:
>
> > >> Failed marshalling 'CONNECTION_REQUEST' protocol message due to:
>
> > >> javax.net.ssl.SSLHandshakeException: Received fatal alert:
>
> > >> bad_certificate
>
> > >>
>
> > >>
>
> > >>
>
> > >> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
>
> > >>
>
> > >>
>
> > >>
>
> > >> Thank you.
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: Mike Thomsen <mi...@gmail.com>>
>
> > >> Sent: Monday, October 15, 2018 9:02 AM
>
> > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> http://nifi.apache.org/docs/nifi-docs/html/administration-guide.h
> > >> tm
>
> > >> l
>
> > >>
>
> > >>
>
> > >>
>
> > >> See the properties that start with "nifi.zookeeper."
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >>
>
> > >> Mike,
>
> > >>
>
> > >>
>
> > >>
>
> > >> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
>
> > >>
>
> > >>
>
> > >>
>
> > >> Thanks again.
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: Mike Thomsen <mi...@gmail.com>>
>
> > >> Sent: Friday, October 12, 2018 10:17 AM
>
> > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
>
> > >>
>
> > >>
>
> > >>
>
> > >> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
>
> > >>
>
> > >>
>
> > >>
>
> > >> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >>
>
> > >> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
>
> > >>
>
> > >>
>
> > >>
>
> > >> 2018-10-12 08:21:35,838 WARN [main]
>
> > >> o.a.nifi.controller.StandardFlowService There is currently no
>
> > >> Cluster Coordinator. This often happens upon restart of NiFi when
>
> > >> running an embedded ZooKeeper. Will register this node to become
>
> > >> the active Cluster Coordinator and will attempt to connect to
>
> > >> cluster again
>
> > >>
>
> > >> 2018-10-12 08:21:35,838 INFO [main]
>
> > >> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> > >> CuratorLeaderElectionManager[stopped=false] Attempted to register
>
> > >> Leader Election for role 'Cluster Coordinator' but this role is
>
> > >> already registered
>
> > >>
>
> > >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
>
> > >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>
> > >>
>
> > >> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
>
> > >> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> > >> org.apache.nifi.controller.leader.election.CuratorLeaderElectionM
> > >> an
>
> > >> ag er$ElectionListener@17900f5b Connection State changed to
>
> > >> SUSPENDED
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: Mike Thomsen <mi...@gmail.com>>
>
> > >> Sent: Friday, October 12, 2018 8:33 AM
>
> > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>> wrote:
>
> > >>
>
> > >> Alexander,
>
> > >>
>
> > >>
>
> > >>
>
> > >> I am pretty sure your problem is here:
>
> > >> nifi.state.management.embedded.zookeeper.start=true
>
> > >>
>
> > >>
>
> > >>
>
> > >> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
>
> > >>
>
> > >>
>
> > >>
>
> > >> What you need to try is these steps:
>
> > >>
>
> > >>
>
> > >>
>
> > >> 1. Set up an external ZooKeeper instance (or set up 3 in a
> > >> quorum;
>
> > >> must be odd numbers)
>
> > >>
>
> > >> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>
> > >>
>
> > >> 3. Restart all of them.
>
> > >>
>
> > >>
>
> > >>
>
> > >> See if that works.
>
> > >>
>
> > >>
>
> > >>
>
> > >> Mike
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >>
>
> > >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
>
> > >>
>
> > >>
>
> > >>
>
> > >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
>
> > >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave
>
> > >> up
>
> > >>
>
> > >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>
> > >> KeeperErrorCode = ConnectionLoss
>
> > >>
>
> > >> at
>
> > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:
> > >> 99
>
> > >> )
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackg
> > >> ro
>
> > >> un
>
> > >> dRetry(CuratorFrameworkImpl.java:728)
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBac
> > >> kg
>
> > >> ro
>
> > >> undOperation(CuratorFrameworkImpl.java:857)
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.background
> > >> Op
>
> > >> er
>
> > >> ationsLoop(CuratorFrameworkImpl.java:809)
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300
> > >> (C
>
> > >> ur
>
> > >> atorFrameworkImpl.java:64)
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(Cur
> > >> at
>
> > >> or
>
> > >> FrameworkImpl.java:267)
>
> > >>
>
> > >> at
> > >> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> > >>
>
> > >> at
>
> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>
> > >> access$201(ScheduledThreadPoolExecutor.java:180)
>
> > >>
>
> > >> at
>
> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>
> > >> run(ScheduledThreadPoolExecutor.java:293)
>
> > >>
>
> > >> at
>
> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
>
> > >> java:1149)
>
> > >>
>
> > >> at
>
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec
> > >> ut
>
> > >> or
>
> > >> .java:624)
>
> > >>
>
> > >> at java.lang.Thread.run(Thread.java:748)
>
> > >>
>
> > >>
>
> > >>
>
> > >> Is there anything else we should check?
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: Nathan Gough <th...@gmail.com>>
>
> > >> Sent: Thursday, October 11, 2018 9:12 AM
>
> > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: ashmeet kandhari <as...@gmail.com>>
>
> > >> Reply-To: <us...@nifi.apache.org>>
>
> > >> Date: Thursday, October 11, 2018 at 9:09 AM
>
> > >> To: <us...@nifi.apache.org>>
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> Hi Alexander,
>
> > >>
>
> > >>
>
> > >>
>
> > >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >>
>
> > >> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: ashmeet kandhari <as...@gmail.com>>
>
> > >> Sent: Thursday, October 11, 2018 4:36 AM
>
> > >> To: users@nifi.apache.org<ma...@nifi.apache.org>
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> Can you see if zookeeper node is up and running and can connect
> > >> to
>
> > >> the nifi nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> > >>
>
> > >> Hello,
>
> > >>
>
> > >>
>
> > >>
>
> > >> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
>
> > >>
>
> > >>
>
> > >>
>
> > >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> > >>
>
> > >> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> > >>
>
> > >> 2018-10-10 13:57:07,748 INFO [main]
>
> > >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties
> > >> from
>
> > >> /opt/nifi-1.7.1/./conf/nifi.properties
>
> > >>
>
> > >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded
> > >> 125
>
> > >> properties
>
> > >>
>
> > >> 2018-10-10 13:57:07,762 INFO [main]
>
> > >> org.apache.nifi.BootstrapListener Started Bootstrap Listener,
>
> > >> Listening for incoming requests on port
>
> > >> 43744
>
> > >>
>
> > >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure
>
> > >> to launch NiFi due to java.net.ConnectException: Connection timed
>
> > >> out (Connection timed out)
>
> > >>
>
> > >> java.net.ConnectException: Connection timed out (Connection timed
>
> > >> out)
>
> > >>
>
> > >> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> > >>
>
> > >> at
>
> > >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.
>
> > >> ja
>
> > >> va:350)
>
> > >>
>
> > >> at
>
> > >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSo
> > >> ck
>
> > >> et
>
> > >> Impl.java:206)
>
> > >>
>
> > >> at
>
> > >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.
> > >> ja
>
> > >> va
>
> > >> :188)
>
> > >>
>
> > >> at
>
> > >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> > >>
>
> > >> at java.net.Socket.connect(Socket.java:589)
>
> > >>
>
> > >> at java.net.Socket.connect(Socket.java:538)
>
> > >>
>
> > >> at
>
> > >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
>
> > >> 100)
>
> > >>
>
> > >> at
>
> > >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83
> > >> )
>
> > >>
>
> > >> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> > >>
>
> > >> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> > >>
>
> > >> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> > >>
>
> > >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
>
> > >>
>
> > >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
>
> > >>
>
> > >>
>
> > >>
>
> > >> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
>
> > >>
>
> > >>
>
> > >>
>
> > >> Thanks in advance,
>
> > >>
>
> > >>
>
> > >>
>
> > >> Alexander
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
If you are getting separate clusters then each node is likely only
using it's own ZooKeeper and therefore doesn't know about the other
node.
In nifi.properties the ZK connect string would need to be something
like nifi-node1-hostname:2181,nifi-node2-hostname:2181 and in
zoo.properties you would need entries for both ZooKeepers:
server.1=nifi-node1-hostname:2888:3888
server.2=nifi-node2-hostname:2888:3888
On Mon, Oct 22, 2018 at 11:28 AM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> I wonder if anyone has run into the same problem when trying to configure composite authentication/authorization (LDAP and local file)? When we use the “stand-alone” authorizers.xml file with the addition of two extra properties
>
>
>
> <property name="Node Identity 1">…
>
> <property name="Node Identity 2">…
>
>
>
> and let ZooKeeper start on one on the nodes, we end up with two one-node clusters, since apparently, the NiFi instances don’t talk to each other, but at least, they come alive…
>
>
>
> From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
> Sent: Friday, October 19, 2018 11:18 AM
> To: users@nifi.apache.org
> Subject: RE: NiFi fails on cluster nodes
>
>
>
> We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
>
>
>
> 2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader Context initialization failed
>
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException: Name is null
>
> at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
>
> at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
>
> at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
>
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
>
> at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
>
> at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
>
> at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
>
> at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
>
> at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
>
> at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876)
>
> at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
>
> at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839)
>
> at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
>
> at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480)
>
> at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442)
>
> at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799)
>
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
>
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
>
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
>
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>
> at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
>
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
>
> at org.eclipse.jetty.server.Server.start(Server.java:452)
>
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
>
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>
> at org.eclipse.jetty.server.Server.doStart(Server.java:419)
>
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:157)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> Caused by: org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException: Name is null
>
> at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:164)
>
> at org.springframework.beans.factory.support.AbstractBeanFactory.evaluateBeanDefinitionString(AbstractBeanFactory.java:1448)
>
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1088)
>
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
>
> at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:659)
>
> ... 48 common frames omitted
>
> Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException: Name is null
>
> at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
>
> at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
>
> at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
>
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
>
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
>
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>
> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:519)
>
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:508)
>
> at org.springframework.security.config.annotation.web.configuration.AutowiredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(AutowiredWebSecurityConfigurersIgnoreParents.java:53)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:113)
>
> at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:129)
>
> at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
>
> at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:347)
>
> at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
>
> at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:120)
>
> at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:262)
>
> at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:161)
>
> ... 52 common frames omitted
>
>
>
> I tried to Google for possible clues, but so far, there hasn’t been any luck…
>
>
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Monday, October 15, 2018 10:27 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
>
>
>
> In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
>
>
>
> Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
>
> On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> >
>
> > Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
>
> >
>
> > -----Original Message-----
>
> > From: Bryan Bende <bb...@gmail.com>
>
> > Sent: Monday, October 15, 2018 9:43 AM
>
> > To: users@nifi.apache.org
>
> > Subject: Re: NiFi fails on cluster nodes
>
> >
>
> > This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
>
> >
>
> > On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> > >
>
> > > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
>
> > >
>
> > > Mike
>
> > >
>
> > > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >>
>
> > >> Hi Mike and Bryan,
>
> > >>
>
> > >>
>
> > >>
>
> > >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
>
> > >>
>
> > >>
>
> > >>
>
> > >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol Request-1]
>
> > >> o.a.nifi.security.util.CertificateUtils The incoming request did
>
> > >> not contain client certificates and thus the DN cannot be extracted.
>
> > >> Check that the other endpoint is providing a complete client
>
> > >> certificate chain
>
> > >>
>
> > >> 2018-10-15 09:19:48,425 INFO [main]
>
> > >> o.a.nifi.controller.StandardFlowService Connecting Node:
>
> > >> 0.0.0.0:8008
>
> > >>
>
> > >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol Request-2]
>
> > >> o.a.nifi.security.util.CertificateUtils The incoming request did
>
> > >> not contain client certificates and thus the DN cannot be extracted.
>
> > >> Check that the other endpoint is providing a complete client
>
> > >> certificate chain
>
> > >>
>
> > >> 2018-10-15 09:19:48,456 WARN [main]
>
> > >> o.a.nifi.controller.StandardFlowService Failed to connect to
>
> > >> cluster due to: org.apache.nifi.cluster.protocol.ProtocolException:
>
> > >> Failed marshalling 'CONNECTION_REQUEST' protocol message due to:
>
> > >> javax.net.ssl.SSLHandshakeException: Received fatal alert:
>
> > >> bad_certificate
>
> > >>
>
> > >>
>
> > >>
>
> > >> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
>
> > >>
>
> > >>
>
> > >>
>
> > >> Thank you.
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: Mike Thomsen <mi...@gmail.com>
>
> > >> Sent: Monday, October 15, 2018 9:02 AM
>
> > >> To: users@nifi.apache.org
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> http://nifi.apache.org/docs/nifi-docs/html/administration-guide.htm
>
> > >> l
>
> > >>
>
> > >>
>
> > >>
>
> > >> See the properties that start with "nifi.zookeeper."
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >>
>
> > >> Mike,
>
> > >>
>
> > >>
>
> > >>
>
> > >> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
>
> > >>
>
> > >>
>
> > >>
>
> > >> Thanks again.
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: Mike Thomsen <mi...@gmail.com>
>
> > >> Sent: Friday, October 12, 2018 10:17 AM
>
> > >> To: users@nifi.apache.org
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
>
> > >>
>
> > >>
>
> > >>
>
> > >> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
>
> > >>
>
> > >>
>
> > >>
>
> > >> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >>
>
> > >> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
>
> > >>
>
> > >>
>
> > >>
>
> > >> 2018-10-12 08:21:35,838 WARN [main]
>
> > >> o.a.nifi.controller.StandardFlowService There is currently no
>
> > >> Cluster Coordinator. This often happens upon restart of NiFi when
>
> > >> running an embedded ZooKeeper. Will register this node to become
>
> > >> the active Cluster Coordinator and will attempt to connect to
>
> > >> cluster again
>
> > >>
>
> > >> 2018-10-12 08:21:35,838 INFO [main]
>
> > >> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> > >> CuratorLeaderElectionManager[stopped=false] Attempted to register
>
> > >> Leader Election for role 'Cluster Coordinator' but this role is
>
> > >> already registered
>
> > >>
>
> > >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
>
> > >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>
> > >>
>
> > >> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
>
> > >> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> > >> org.apache.nifi.controller.leader.election.CuratorLeaderElectionMan
>
> > >> ag er$ElectionListener@17900f5b Connection State changed to
>
> > >> SUSPENDED
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: Mike Thomsen <mi...@gmail.com>
>
> > >> Sent: Friday, October 12, 2018 8:33 AM
>
> > >> To: users@nifi.apache.org
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> > >>
>
> > >> Alexander,
>
> > >>
>
> > >>
>
> > >>
>
> > >> I am pretty sure your problem is here:
>
> > >> nifi.state.management.embedded.zookeeper.start=true
>
> > >>
>
> > >>
>
> > >>
>
> > >> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
>
> > >>
>
> > >>
>
> > >>
>
> > >> What you need to try is these steps:
>
> > >>
>
> > >>
>
> > >>
>
> > >> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum;
>
> > >> must be odd numbers)
>
> > >>
>
> > >> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>
> > >>
>
> > >> 3. Restart all of them.
>
> > >>
>
> > >>
>
> > >>
>
> > >> See if that works.
>
> > >>
>
> > >>
>
> > >>
>
> > >> Mike
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >>
>
> > >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
>
> > >>
>
> > >>
>
> > >>
>
> > >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
>
> > >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave
>
> > >> up
>
> > >>
>
> > >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>
> > >> KeeperErrorCode = ConnectionLoss
>
> > >>
>
> > >> at
>
> > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99
>
> > >> )
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgro
>
> > >> un
>
> > >> dRetry(CuratorFrameworkImpl.java:728)
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackg
>
> > >> ro
>
> > >> undOperation(CuratorFrameworkImpl.java:857)
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOp
>
> > >> er
>
> > >> ationsLoop(CuratorFrameworkImpl.java:809)
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(C
>
> > >> ur
>
> > >> atorFrameworkImpl.java:64)
>
> > >>
>
> > >> at
>
> > >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(Curat
>
> > >> or
>
> > >> FrameworkImpl.java:267)
>
> > >>
>
> > >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> > >>
>
> > >> at
>
> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>
> > >> access$201(ScheduledThreadPoolExecutor.java:180)
>
> > >>
>
> > >> at
>
> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>
> > >> run(ScheduledThreadPoolExecutor.java:293)
>
> > >>
>
> > >> at
>
> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
>
> > >> java:1149)
>
> > >>
>
> > >> at
>
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
>
> > >> or
>
> > >> .java:624)
>
> > >>
>
> > >> at java.lang.Thread.run(Thread.java:748)
>
> > >>
>
> > >>
>
> > >>
>
> > >> Is there anything else we should check?
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: Nathan Gough <th...@gmail.com>
>
> > >> Sent: Thursday, October 11, 2018 9:12 AM
>
> > >> To: users@nifi.apache.org
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: ashmeet kandhari <as...@gmail.com>
>
> > >> Reply-To: <us...@nifi.apache.org>
>
> > >> Date: Thursday, October 11, 2018 at 9:09 AM
>
> > >> To: <us...@nifi.apache.org>
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> Hi Alexander,
>
> > >>
>
> > >>
>
> > >>
>
> > >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >>
>
> > >> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
>
> > >>
>
> > >>
>
> > >>
>
> > >> From: ashmeet kandhari <as...@gmail.com>
>
> > >> Sent: Thursday, October 11, 2018 4:36 AM
>
> > >> To: users@nifi.apache.org
>
> > >> Subject: Re: NiFi fails on cluster nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> Can you see if zookeeper node is up and running and can connect to
>
> > >> the nifi nodes
>
> > >>
>
> > >>
>
> > >>
>
> > >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> > >>
>
> > >> Hello,
>
> > >>
>
> > >>
>
> > >>
>
> > >> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
>
> > >>
>
> > >>
>
> > >>
>
> > >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> > >>
>
> > >> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> > >>
>
> > >> 2018-10-10 13:57:07,748 INFO [main]
>
> > >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
>
> > >> /opt/nifi-1.7.1/./conf/nifi.properties
>
> > >>
>
> > >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
>
> > >> properties
>
> > >>
>
> > >> 2018-10-10 13:57:07,762 INFO [main]
>
> > >> org.apache.nifi.BootstrapListener Started Bootstrap Listener,
>
> > >> Listening for incoming requests on port
>
> > >> 43744
>
> > >>
>
> > >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure
>
> > >> to launch NiFi due to java.net.ConnectException: Connection timed
>
> > >> out (Connection timed out)
>
> > >>
>
> > >> java.net.ConnectException: Connection timed out (Connection timed
>
> > >> out)
>
> > >>
>
> > >> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> > >>
>
> > >> at
>
> > >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.
>
> > >> ja
>
> > >> va:350)
>
> > >>
>
> > >> at
>
> > >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSock
>
> > >> et
>
> > >> Impl.java:206)
>
> > >>
>
> > >> at
>
> > >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.ja
>
> > >> va
>
> > >> :188)
>
> > >>
>
> > >> at
>
> > >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> > >>
>
> > >> at java.net.Socket.connect(Socket.java:589)
>
> > >>
>
> > >> at java.net.Socket.connect(Socket.java:538)
>
> > >>
>
> > >> at
>
> > >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
>
> > >> 100)
>
> > >>
>
> > >> at
>
> > >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>
> > >>
>
> > >> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> > >>
>
> > >> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> > >>
>
> > >> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> > >>
>
> > >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
>
> > >>
>
> > >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
>
> > >>
>
> > >>
>
> > >>
>
> > >> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
>
> > >>
>
> > >>
>
> > >>
>
> > >> Thanks in advance,
>
> > >>
>
> > >>
>
> > >>
>
> > >> Alexander
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
I wonder if anyone has run into the same problem when trying to configure composite authentication/authorization (LDAP and local file)? When we use the “stand-alone” authorizers.xml file with the addition of two extra properties
<property name="Node Identity 1">…
<property name="Node Identity 2">…
and let ZooKeeper start on one on the nodes, we end up with two one-node clusters, since apparently, the NiFi instances don’t talk to each other, but at least, they come alive…
From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
Sent: Friday, October 19, 2018 11:18 AM
To: users@nifi.apache.org
Subject: RE: NiFi fails on cluster nodes
We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#initial-admin-identity> as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException: Name is null
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838)
at org.apache.nifi.NiFi.<init>(NiFi.java:157)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
Caused by: org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException: Name is null
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:164)
at org.springframework.beans.factory.support.AbstractBeanFactory.evaluateBeanDefinitionString(AbstractBeanFactory.java:1448)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1088)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:659)
... 48 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException: Name is null
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:519)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:508)
at org.springframework.security.config.annotation.web.configuration.AutowiredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(AutowiredWebSecurityConfigurersIgnoreParents.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:113)
at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:129)
at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:347)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:120)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:262)
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:161)
... 52 common frames omitted
I tried to Google for possible clues, but so far, there hasn’t been any luck…
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>>
Sent: Monday, October 15, 2018 10:27 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Monday, October 15, 2018 9:43 AM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
> This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
>
> On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com>> wrote:
> >
> > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
> >
> > Mike
> >
> > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> Hi Mike and Bryan,
> >>
> >>
> >>
> >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
> >>
> >>
> >>
> >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol Request-1]
> >> o.a.nifi.security.util.CertificateUtils The incoming request did
> >> not contain client certificates and thus the DN cannot be extracted.
> >> Check that the other endpoint is providing a complete client
> >> certificate chain
> >>
> >> 2018-10-15 09:19:48,425 INFO [main]
> >> o.a.nifi.controller.StandardFlowService Connecting Node:
> >> 0.0.0.0:8008
> >>
> >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol Request-2]
> >> o.a.nifi.security.util.CertificateUtils The incoming request did
> >> not contain client certificates and thus the DN cannot be extracted.
> >> Check that the other endpoint is providing a complete client
> >> certificate chain
> >>
> >> 2018-10-15 09:19:48,456 WARN [main]
> >> o.a.nifi.controller.StandardFlowService Failed to connect to
> >> cluster due to: org.apache.nifi.cluster.protocol.ProtocolException:
> >> Failed marshalling 'CONNECTION_REQUEST' protocol message due to:
> >> javax.net.ssl.SSLHandshakeException: Received fatal alert:
> >> bad_certificate
> >>
> >>
> >>
> >> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
> >>
> >>
> >>
> >> Thank you.
> >>
> >>
> >>
> >> From: Mike Thomsen <mi...@gmail.com>>
> >> Sent: Monday, October 15, 2018 9:02 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> http://nifi.apache.org/docs/nifi-docs/html/administration-guide.htm
> >> l
> >>
> >>
> >>
> >> See the properties that start with "nifi.zookeeper."
> >>
> >>
> >>
> >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> Mike,
> >>
> >>
> >>
> >> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
> >>
> >>
> >>
> >> Thanks again.
> >>
> >>
> >>
> >> From: Mike Thomsen <mi...@gmail.com>>
> >> Sent: Friday, October 12, 2018 10:17 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
> >>
> >>
> >>
> >> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
> >>
> >>
> >>
> >> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
> >>
> >>
> >>
> >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
> >>
> >>
> >>
> >> 2018-10-12 08:21:35,838 WARN [main]
> >> o.a.nifi.controller.StandardFlowService There is currently no
> >> Cluster Coordinator. This often happens upon restart of NiFi when
> >> running an embedded ZooKeeper. Will register this node to become
> >> the active Cluster Coordinator and will attempt to connect to
> >> cluster again
> >>
> >> 2018-10-12 08:21:35,838 INFO [main]
> >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >> CuratorLeaderElectionManager[stopped=false] Attempted to register
> >> Leader Election for role 'Cluster Coordinator' but this role is
> >> already registered
> >>
> >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
> >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
> >>
> >> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
> >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >> org.apache.nifi.controller.leader.election.CuratorLeaderElectionMan
> >> ag er$ElectionListener@17900f5b Connection State changed to
> >> SUSPENDED
> >>
> >>
> >>
> >> From: Mike Thomsen <mi...@gmail.com>>
> >> Sent: Friday, October 12, 2018 8:33 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
> >>
> >>
> >>
> >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>> wrote:
> >>
> >> Alexander,
> >>
> >>
> >>
> >> I am pretty sure your problem is here:
> >> nifi.state.management.embedded.zookeeper.start=true
> >>
> >>
> >>
> >> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
> >>
> >>
> >>
> >> What you need to try is these steps:
> >>
> >>
> >>
> >> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum;
> >> must be odd numbers)
> >>
> >> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
> >>
> >> 3. Restart all of them.
> >>
> >>
> >>
> >> See if that works.
> >>
> >>
> >>
> >> Mike
> >>
> >>
> >>
> >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
> >>
> >>
> >>
> >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
> >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave
> >> up
> >>
> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> KeeperErrorCode = ConnectionLoss
> >>
> >> at
> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99
> >> )
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgro
> >> un
> >> dRetry(CuratorFrameworkImpl.java:728)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackg
> >> ro
> >> undOperation(CuratorFrameworkImpl.java:857)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOp
> >> er
> >> ationsLoop(CuratorFrameworkImpl.java:809)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(C
> >> ur
> >> atorFrameworkImpl.java:64)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(Curat
> >> or
> >> FrameworkImpl.java:267)
> >>
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >>
> >> at
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >> access$201(ScheduledThreadPoolExecutor.java:180)
> >>
> >> at
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >> run(ScheduledThreadPoolExecutor.java:293)
> >>
> >> at
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> >> java:1149)
> >>
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
> >> or
> >> .java:624)
> >>
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >>
> >>
> >> Is there anything else we should check?
> >>
> >>
> >>
> >> From: Nathan Gough <th...@gmail.com>>
> >> Sent: Thursday, October 11, 2018 9:12 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
> >>
> >>
> >>
> >> From: ashmeet kandhari <as...@gmail.com>>
> >> Reply-To: <us...@nifi.apache.org>>
> >> Date: Thursday, October 11, 2018 at 9:09 AM
> >> To: <us...@nifi.apache.org>>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> Hi Alexander,
> >>
> >>
> >>
> >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
> >>
> >>
> >>
> >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
> >>
> >>
> >>
> >> From: ashmeet kandhari <as...@gmail.com>>
> >> Sent: Thursday, October 11, 2018 4:36 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> Can you see if zookeeper node is up and running and can connect to
> >> the nifi nodes
> >>
> >>
> >>
> >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> Hello,
> >>
> >>
> >>
> >> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
> >>
> >>
> >>
> >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
> >>
> >> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
> >>
> >> 2018-10-10 13:57:07,748 INFO [main]
> >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
> >> /opt/nifi-1.7.1/./conf/nifi.properties
> >>
> >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
> >> properties
> >>
> >> 2018-10-10 13:57:07,762 INFO [main]
> >> org.apache.nifi.BootstrapListener Started Bootstrap Listener,
> >> Listening for incoming requests on port
> >> 43744
> >>
> >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure
> >> to launch NiFi due to java.net.ConnectException: Connection timed
> >> out (Connection timed out)
> >>
> >> java.net.ConnectException: Connection timed out (Connection timed
> >> out)
> >>
> >> at java.net.PlainSocketImpl.socketConnect(Native Method)
> >>
> >> at
> >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.
> >> ja
> >> va:350)
> >>
> >> at
> >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSock
> >> et
> >> Impl.java:206)
> >>
> >> at
> >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.ja
> >> va
> >> :188)
> >>
> >> at
> >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >>
> >> at java.net.Socket.connect(Socket.java:589)
> >>
> >> at java.net.Socket.connect(Socket.java:538)
> >>
> >> at
> >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
> >> 100)
> >>
> >> at
> >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
> >>
> >> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
> >>
> >> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> >>
> >> at org.apache.nifi.NiFi.main(NiFi.java:292)
> >>
> >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
> >>
> >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
> >>
> >>
> >>
> >> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
> >>
> >>
> >>
> >> Thanks in advance,
> >>
> >>
> >>
> >> Alexander
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
We have managed to get past that error by installing the CA cert in the truststore. So, we can get a one-node cluster up and running. In order to add another node, I edited the authorizers.xml file, basically, using the “example composite implementation loading users and groups from LDAP and a local file” from the Admin guide<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#initial-admin-identity> as a template. When I re-started the node running ZooKeeper, though, it crashed with the following error written into the nifi-app.log file:
2018-10-19 08:09:26,992 ERROR [main] o.s.web.context.ContextLoader Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException: Name is null
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838)
at org.apache.nifi.NiFi.<init>(NiFi.java:157)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
Caused by: org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException: Name is null
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:164)
at org.springframework.beans.factory.support.AbstractBeanFactory.evaluateBeanDefinitionString(AbstractBeanFactory.java:1448)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1088)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:659)
... 48 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'authorizer' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is java.lang.NullPointerException: Name is null
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:519)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:508)
at org.springframework.security.config.annotation.web.configuration.AutowiredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(AutowiredWebSecurityConfigurersIgnoreParents.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:113)
at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:129)
at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:347)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:120)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:262)
at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:161)
... 52 common frames omitted
I tried to Google for possible clues, but so far, there hasn’t been any luck…
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Monday, October 15, 2018 10:27 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
I'm not really sure, the error message is indicating that either a certificate was not sent during cluster communications, or possibly the cert was not valid/trusted.
In this case since it is only 1 node, it is the same node talking back to itself, so the only parts involved here are the keystore and truststore of that node, and the config in nifi.properties.
Maybe your truststore is not setup correctly to trust certs signed by the CA that created the server cert?
On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
>
> Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>>
> Sent: Monday, October 15, 2018 9:43 AM
> To: users@nifi.apache.org<ma...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
> This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
>
> On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com>> wrote:
> >
> > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
> >
> > Mike
> >
> > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> Hi Mike and Bryan,
> >>
> >>
> >>
> >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
> >>
> >>
> >>
> >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol Request-1]
> >> o.a.nifi.security.util.CertificateUtils The incoming request did
> >> not contain client certificates and thus the DN cannot be extracted.
> >> Check that the other endpoint is providing a complete client
> >> certificate chain
> >>
> >> 2018-10-15 09:19:48,425 INFO [main]
> >> o.a.nifi.controller.StandardFlowService Connecting Node:
> >> 0.0.0.0:8008
> >>
> >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol Request-2]
> >> o.a.nifi.security.util.CertificateUtils The incoming request did
> >> not contain client certificates and thus the DN cannot be extracted.
> >> Check that the other endpoint is providing a complete client
> >> certificate chain
> >>
> >> 2018-10-15 09:19:48,456 WARN [main]
> >> o.a.nifi.controller.StandardFlowService Failed to connect to
> >> cluster due to: org.apache.nifi.cluster.protocol.ProtocolException:
> >> Failed marshalling 'CONNECTION_REQUEST' protocol message due to:
> >> javax.net.ssl.SSLHandshakeException: Received fatal alert:
> >> bad_certificate
> >>
> >>
> >>
> >> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
> >>
> >>
> >>
> >> Thank you.
> >>
> >>
> >>
> >> From: Mike Thomsen <mi...@gmail.com>>
> >> Sent: Monday, October 15, 2018 9:02 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> http://nifi.apache.org/docs/nifi-docs/html/administration-guide.htm
> >> l
> >>
> >>
> >>
> >> See the properties that start with "nifi.zookeeper."
> >>
> >>
> >>
> >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> Mike,
> >>
> >>
> >>
> >> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
> >>
> >>
> >>
> >> Thanks again.
> >>
> >>
> >>
> >> From: Mike Thomsen <mi...@gmail.com>>
> >> Sent: Friday, October 12, 2018 10:17 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
> >>
> >>
> >>
> >> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
> >>
> >>
> >>
> >> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
> >>
> >>
> >>
> >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
> >>
> >>
> >>
> >> 2018-10-12 08:21:35,838 WARN [main]
> >> o.a.nifi.controller.StandardFlowService There is currently no
> >> Cluster Coordinator. This often happens upon restart of NiFi when
> >> running an embedded ZooKeeper. Will register this node to become
> >> the active Cluster Coordinator and will attempt to connect to
> >> cluster again
> >>
> >> 2018-10-12 08:21:35,838 INFO [main]
> >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >> CuratorLeaderElectionManager[stopped=false] Attempted to register
> >> Leader Election for role 'Cluster Coordinator' but this role is
> >> already registered
> >>
> >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
> >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
> >>
> >> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
> >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >> org.apache.nifi.controller.leader.election.CuratorLeaderElectionMan
> >> ag er$ElectionListener@17900f5b Connection State changed to
> >> SUSPENDED
> >>
> >>
> >>
> >> From: Mike Thomsen <mi...@gmail.com>>
> >> Sent: Friday, October 12, 2018 8:33 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
> >>
> >>
> >>
> >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>> wrote:
> >>
> >> Alexander,
> >>
> >>
> >>
> >> I am pretty sure your problem is here:
> >> nifi.state.management.embedded.zookeeper.start=true
> >>
> >>
> >>
> >> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
> >>
> >>
> >>
> >> What you need to try is these steps:
> >>
> >>
> >>
> >> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum;
> >> must be odd numbers)
> >>
> >> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
> >>
> >> 3. Restart all of them.
> >>
> >>
> >>
> >> See if that works.
> >>
> >>
> >>
> >> Mike
> >>
> >>
> >>
> >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
> >>
> >>
> >>
> >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
> >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave
> >> up
> >>
> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> KeeperErrorCode = ConnectionLoss
> >>
> >> at
> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99
> >> )
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgro
> >> un
> >> dRetry(CuratorFrameworkImpl.java:728)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackg
> >> ro
> >> undOperation(CuratorFrameworkImpl.java:857)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOp
> >> er
> >> ationsLoop(CuratorFrameworkImpl.java:809)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(C
> >> ur
> >> atorFrameworkImpl.java:64)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(Curat
> >> or
> >> FrameworkImpl.java:267)
> >>
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >>
> >> at
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >> access$201(ScheduledThreadPoolExecutor.java:180)
> >>
> >> at
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >> run(ScheduledThreadPoolExecutor.java:293)
> >>
> >> at
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> >> java:1149)
> >>
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
> >> or
> >> .java:624)
> >>
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >>
> >>
> >> Is there anything else we should check?
> >>
> >>
> >>
> >> From: Nathan Gough <th...@gmail.com>>
> >> Sent: Thursday, October 11, 2018 9:12 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
> >>
> >>
> >>
> >> From: ashmeet kandhari <as...@gmail.com>>
> >> Reply-To: <us...@nifi.apache.org>>
> >> Date: Thursday, October 11, 2018 at 9:09 AM
> >> To: <us...@nifi.apache.org>>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> Hi Alexander,
> >>
> >>
> >>
> >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
> >>
> >>
> >>
> >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
> >>
> >>
> >>
> >> From: ashmeet kandhari <as...@gmail.com>>
> >> Sent: Thursday, October 11, 2018 4:36 AM
> >> To: users@nifi.apache.org<ma...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> Can you see if zookeeper node is up and running and can connect to
> >> the nifi nodes
> >>
> >>
> >>
> >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
> >>
> >> Hello,
> >>
> >>
> >>
> >> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
> >>
> >>
> >>
> >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
> >>
> >> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
> >>
> >> 2018-10-10 13:57:07,748 INFO [main]
> >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
> >> /opt/nifi-1.7.1/./conf/nifi.properties
> >>
> >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
> >> properties
> >>
> >> 2018-10-10 13:57:07,762 INFO [main]
> >> org.apache.nifi.BootstrapListener Started Bootstrap Listener,
> >> Listening for incoming requests on port
> >> 43744
> >>
> >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure
> >> to launch NiFi due to java.net.ConnectException: Connection timed
> >> out (Connection timed out)
> >>
> >> java.net.ConnectException: Connection timed out (Connection timed
> >> out)
> >>
> >> at java.net.PlainSocketImpl.socketConnect(Native Method)
> >>
> >> at
> >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.
> >> ja
> >> va:350)
> >>
> >> at
> >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSock
> >> et
> >> Impl.java:206)
> >>
> >> at
> >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.ja
> >> va
> >> :188)
> >>
> >> at
> >> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >>
> >> at java.net.Socket.connect(Socket.java:589)
> >>
> >> at java.net.Socket.connect(Socket.java:538)
> >>
> >> at
> >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
> >> 100)
> >>
> >> at
> >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
> >>
> >> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
> >>
> >> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> >>
> >> at org.apache.nifi.NiFi.main(NiFi.java:292)
> >>
> >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
> >>
> >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
> >>
> >>
> >>
> >> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
> >>
> >>
> >>
> >> Thanks in advance,
> >>
> >>
> >>
> >> Alexander
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
I'm not really sure, the error message is indicating that either a
certificate was not sent during cluster communications, or possibly
the cert was not valid/trusted.
In this case since it is only 1 node, it is the same node talking back
to itself, so the only parts involved here are the keystore and
truststore of that node, and the config in nifi.properties.
Maybe your truststore is not setup correctly to trust certs signed by
the CA that created the server cert?
On Mon, Oct 15, 2018 at 9:53 AM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
>
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: Monday, October 15, 2018 9:43 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
> This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
>
> On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com> wrote:
> >
> > Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
> >
> > Mike
> >
> > On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >>
> >> Hi Mike and Bryan,
> >>
> >>
> >>
> >> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
> >>
> >>
> >>
> >> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol Request-1]
> >> o.a.nifi.security.util.CertificateUtils The incoming request did not
> >> contain client certificates and thus the DN cannot be extracted.
> >> Check that the other endpoint is providing a complete client
> >> certificate chain
> >>
> >> 2018-10-15 09:19:48,425 INFO [main]
> >> o.a.nifi.controller.StandardFlowService Connecting Node: 0.0.0.0:8008
> >>
> >> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol Request-2]
> >> o.a.nifi.security.util.CertificateUtils The incoming request did not
> >> contain client certificates and thus the DN cannot be extracted.
> >> Check that the other endpoint is providing a complete client
> >> certificate chain
> >>
> >> 2018-10-15 09:19:48,456 WARN [main]
> >> o.a.nifi.controller.StandardFlowService Failed to connect to cluster
> >> due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed
> >> marshalling 'CONNECTION_REQUEST' protocol message due to:
> >> javax.net.ssl.SSLHandshakeException: Received fatal alert:
> >> bad_certificate
> >>
> >>
> >>
> >> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
> >>
> >>
> >>
> >> Thank you.
> >>
> >>
> >>
> >> From: Mike Thomsen <mi...@gmail.com>
> >> Sent: Monday, October 15, 2018 9:02 AM
> >> To: users@nifi.apache.org
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
> >>
> >>
> >>
> >> See the properties that start with "nifi.zookeeper."
> >>
> >>
> >>
> >> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >>
> >> Mike,
> >>
> >>
> >>
> >> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
> >>
> >>
> >>
> >> Thanks again.
> >>
> >>
> >>
> >> From: Mike Thomsen <mi...@gmail.com>
> >> Sent: Friday, October 12, 2018 10:17 AM
> >> To: users@nifi.apache.org
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
> >>
> >>
> >>
> >> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
> >>
> >>
> >>
> >> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
> >>
> >>
> >>
> >> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >>
> >> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
> >>
> >>
> >>
> >> 2018-10-12 08:21:35,838 WARN [main]
> >> o.a.nifi.controller.StandardFlowService There is currently no Cluster
> >> Coordinator. This often happens upon restart of NiFi when running an
> >> embedded ZooKeeper. Will register this node to become the active
> >> Cluster Coordinator and will attempt to connect to cluster again
> >>
> >> 2018-10-12 08:21:35,838 INFO [main]
> >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >> CuratorLeaderElectionManager[stopped=false] Attempted to register
> >> Leader Election for role 'Cluster Coordinator' but this role is
> >> already registered
> >>
> >> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
> >> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
> >>
> >> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
> >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManag
> >> er$ElectionListener@17900f5b Connection State changed to SUSPENDED
> >>
> >>
> >>
> >> From: Mike Thomsen <mi...@gmail.com>
> >> Sent: Friday, October 12, 2018 8:33 AM
> >> To: users@nifi.apache.org
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
> >>
> >>
> >>
> >> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com> wrote:
> >>
> >> Alexander,
> >>
> >>
> >>
> >> I am pretty sure your problem is here:
> >> nifi.state.management.embedded.zookeeper.start=true
> >>
> >>
> >>
> >> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
> >>
> >>
> >>
> >> What you need to try is these steps:
> >>
> >>
> >>
> >> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum;
> >> must be odd numbers)
> >>
> >> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
> >>
> >> 3. Restart all of them.
> >>
> >>
> >>
> >> See if that works.
> >>
> >>
> >>
> >> Mike
> >>
> >>
> >>
> >> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >>
> >> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
> >>
> >>
> >>
> >> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
> >> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
> >>
> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> KeeperErrorCode = ConnectionLoss
> >>
> >> at
> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroun
> >> dRetry(CuratorFrameworkImpl.java:728)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgro
> >> undOperation(CuratorFrameworkImpl.java:857)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOper
> >> ationsLoop(CuratorFrameworkImpl.java:809)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(Cur
> >> atorFrameworkImpl.java:64)
> >>
> >> at
> >> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(Curator
> >> FrameworkImpl.java:267)
> >>
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >>
> >> at
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >> access$201(ScheduledThreadPoolExecutor.java:180)
> >>
> >> at
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> >> run(ScheduledThreadPoolExecutor.java:293)
> >>
> >> at
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> >> java:1149)
> >>
> >> at
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
> >> .java:624)
> >>
> >> at java.lang.Thread.run(Thread.java:748)
> >>
> >>
> >>
> >> Is there anything else we should check?
> >>
> >>
> >>
> >> From: Nathan Gough <th...@gmail.com>
> >> Sent: Thursday, October 11, 2018 9:12 AM
> >> To: users@nifi.apache.org
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
> >>
> >>
> >>
> >> From: ashmeet kandhari <as...@gmail.com>
> >> Reply-To: <us...@nifi.apache.org>
> >> Date: Thursday, October 11, 2018 at 9:09 AM
> >> To: <us...@nifi.apache.org>
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> Hi Alexander,
> >>
> >>
> >>
> >> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
> >>
> >>
> >>
> >> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >>
> >> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
> >>
> >>
> >>
> >> From: ashmeet kandhari <as...@gmail.com>
> >> Sent: Thursday, October 11, 2018 4:36 AM
> >> To: users@nifi.apache.org
> >> Subject: Re: NiFi fails on cluster nodes
> >>
> >>
> >>
> >> Can you see if zookeeper node is up and running and can connect to
> >> the nifi nodes
> >>
> >>
> >>
> >> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
> >>
> >> Hello,
> >>
> >>
> >>
> >> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
> >>
> >>
> >>
> >> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
> >>
> >> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
> >>
> >> 2018-10-10 13:57:07,748 INFO [main]
> >> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
> >> /opt/nifi-1.7.1/./conf/nifi.properties
> >>
> >> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
> >> properties
> >>
> >> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener
> >> Started Bootstrap Listener, Listening for incoming requests on port
> >> 43744
> >>
> >> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to
> >> launch NiFi due to java.net.ConnectException: Connection timed out
> >> (Connection timed out)
> >>
> >> java.net.ConnectException: Connection timed out (Connection timed
> >> out)
> >>
> >> at java.net.PlainSocketImpl.socketConnect(Native Method)
> >>
> >> at
> >> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
> >> va:350)
> >>
> >> at
> >> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
> >> Impl.java:206)
> >>
> >> at
> >> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
> >> :188)
> >>
> >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >>
> >> at java.net.Socket.connect(Socket.java:589)
> >>
> >> at java.net.Socket.connect(Socket.java:538)
> >>
> >> at
> >> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
> >> 100)
> >>
> >> at
> >> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
> >>
> >> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
> >>
> >> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
> >>
> >> at org.apache.nifi.NiFi.main(NiFi.java:292)
> >>
> >> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
> >>
> >> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
> >>
> >>
> >>
> >> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
> >>
> >>
> >>
> >> Thanks in advance,
> >>
> >>
> >>
> >> Alexander
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Yes, 'nifi.cluster.protocol.is.secure' is set to 'true', since otherwise, NiFi would require values for 'nifi.web.http.host' and 'nifi.web.http.port'. We have a cert that is used to serve HTTPS requests to the NiFi web UI, and it works just fine.
-----Original Message-----
From: Bryan Bende <bb...@gmail.com>
Sent: Monday, October 15, 2018 9:43 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
This is not related to ZooKeeper... I think you are missing something related to TLS/SSL configuration, maybe you set cluster protocol to be secure, but then you didn't configure NiFi with a keystore/truststore?
On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
>
> Mike
>
> On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> Hi Mike and Bryan,
>>
>>
>>
>> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
>>
>>
>>
>> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol Request-1]
>> o.a.nifi.security.util.CertificateUtils The incoming request did not
>> contain client certificates and thus the DN cannot be extracted.
>> Check that the other endpoint is providing a complete client
>> certificate chain
>>
>> 2018-10-15 09:19:48,425 INFO [main]
>> o.a.nifi.controller.StandardFlowService Connecting Node: 0.0.0.0:8008
>>
>> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol Request-2]
>> o.a.nifi.security.util.CertificateUtils The incoming request did not
>> contain client certificates and thus the DN cannot be extracted.
>> Check that the other endpoint is providing a complete client
>> certificate chain
>>
>> 2018-10-15 09:19:48,456 WARN [main]
>> o.a.nifi.controller.StandardFlowService Failed to connect to cluster
>> due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed
>> marshalling 'CONNECTION_REQUEST' protocol message due to:
>> javax.net.ssl.SSLHandshakeException: Received fatal alert:
>> bad_certificate
>>
>>
>>
>> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
>>
>>
>>
>> Thank you.
>>
>>
>>
>> From: Mike Thomsen <mi...@gmail.com>
>> Sent: Monday, October 15, 2018 9:02 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
>>
>>
>>
>> See the properties that start with "nifi.zookeeper."
>>
>>
>>
>> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> Mike,
>>
>>
>>
>> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
>>
>>
>>
>> Thanks again.
>>
>>
>>
>> From: Mike Thomsen <mi...@gmail.com>
>> Sent: Friday, October 12, 2018 10:17 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
>>
>>
>>
>> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
>>
>>
>>
>> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
>>
>>
>>
>> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
>>
>>
>>
>> 2018-10-12 08:21:35,838 WARN [main]
>> o.a.nifi.controller.StandardFlowService There is currently no Cluster
>> Coordinator. This often happens upon restart of NiFi when running an
>> embedded ZooKeeper. Will register this node to become the active
>> Cluster Coordinator and will attempt to connect to cluster again
>>
>> 2018-10-12 08:21:35,838 INFO [main]
>> o.a.n.c.l.e.CuratorLeaderElectionManager
>> CuratorLeaderElectionManager[stopped=false] Attempted to register
>> Leader Election for role 'Cluster Coordinator' but this role is
>> already registered
>>
>> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
>> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>>
>> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
>> o.a.n.c.l.e.CuratorLeaderElectionManager
>> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManag
>> er$ElectionListener@17900f5b Connection State changed to SUSPENDED
>>
>>
>>
>> From: Mike Thomsen <mi...@gmail.com>
>> Sent: Friday, October 12, 2018 8:33 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
>>
>>
>>
>> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com> wrote:
>>
>> Alexander,
>>
>>
>>
>> I am pretty sure your problem is here:
>> nifi.state.management.embedded.zookeeper.start=true
>>
>>
>>
>> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
>>
>>
>>
>> What you need to try is these steps:
>>
>>
>>
>> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum;
>> must be odd numbers)
>>
>> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>>
>> 3. Restart all of them.
>>
>>
>>
>> See if that works.
>>
>>
>>
>> Mike
>>
>>
>>
>> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
>>
>>
>>
>> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
>> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
>>
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss
>>
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroun
>> dRetry(CuratorFrameworkImpl.java:728)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgro
>> undOperation(CuratorFrameworkImpl.java:857)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOper
>> ationsLoop(CuratorFrameworkImpl.java:809)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(Cur
>> atorFrameworkImpl.java:64)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(Curator
>> FrameworkImpl.java:267)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>> access$201(ScheduledThreadPoolExecutor.java:180)
>>
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
>> run(ScheduledThreadPoolExecutor.java:293)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
>> java:1149)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
>> .java:624)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>> Is there anything else we should check?
>>
>>
>>
>> From: Nathan Gough <th...@gmail.com>
>> Sent: Thursday, October 11, 2018 9:12 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
>>
>>
>>
>> From: ashmeet kandhari <as...@gmail.com>
>> Reply-To: <us...@nifi.apache.org>
>> Date: Thursday, October 11, 2018 at 9:09 AM
>> To: <us...@nifi.apache.org>
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> Hi Alexander,
>>
>>
>>
>> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
>>
>>
>>
>> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
>>
>>
>>
>> From: ashmeet kandhari <as...@gmail.com>
>> Sent: Thursday, October 11, 2018 4:36 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> Can you see if zookeeper node is up and running and can connect to
>> the nifi nodes
>>
>>
>>
>> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> Hello,
>>
>>
>>
>> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
>>
>>
>>
>> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>>
>> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>>
>> 2018-10-10 13:57:07,748 INFO [main]
>> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
>> /opt/nifi-1.7.1/./conf/nifi.properties
>>
>> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
>> properties
>>
>> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener
>> Started Bootstrap Listener, Listening for incoming requests on port
>> 43744
>>
>> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to
>> launch NiFi due to java.net.ConnectException: Connection timed out
>> (Connection timed out)
>>
>> java.net.ConnectException: Connection timed out (Connection timed
>> out)
>>
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
>> va:350)
>>
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
>> Impl.java:206)
>>
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
>> :188)
>>
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>
>> at java.net.Socket.connect(Socket.java:589)
>>
>> at java.net.Socket.connect(Socket.java:538)
>>
>> at
>> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:
>> 100)
>>
>> at
>> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>>
>> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>>
>> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>>
>> at org.apache.nifi.NiFi.main(NiFi.java:292)
>>
>> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
>>
>> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
>>
>>
>>
>> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
>>
>>
>>
>> Thanks in advance,
>>
>>
>>
>> Alexander
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
This is not related to ZooKeeper... I think you are missing something
related to TLS/SSL configuration, maybe you set cluster protocol to be
secure, but then you didn't configure NiFi with a keystore/truststore?
On Mon, Oct 15, 2018 at 9:41 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> Not sure what's going on here, but NiFi does not require a cert to setup ZooKeeper.
>
> Mike
>
> On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> Hi Mike and Bryan,
>>
>>
>>
>> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
>>
>>
>>
>> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol Request-1] o.a.nifi.security.util.CertificateUtils The incoming request did not contain client certificates and thus the DN cannot be extracted. Check that the other endpoint is providing a complete client certificate chain
>>
>> 2018-10-15 09:19:48,425 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: 0.0.0.0:8008
>>
>> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol Request-2] o.a.nifi.security.util.CertificateUtils The incoming request did not contain client certificates and thus the DN cannot be extracted. Check that the other endpoint is providing a complete client certificate chain
>>
>> 2018-10-15 09:19:48,456 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
>>
>>
>>
>> It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
>>
>>
>>
>> Thank you.
>>
>>
>>
>> From: Mike Thomsen <mi...@gmail.com>
>> Sent: Monday, October 15, 2018 9:02 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
>>
>>
>>
>> See the properties that start with "nifi.zookeeper."
>>
>>
>>
>> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> Mike,
>>
>>
>>
>> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
>>
>>
>>
>> Thanks again.
>>
>>
>>
>> From: Mike Thomsen <mi...@gmail.com>
>> Sent: Friday, October 12, 2018 10:17 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
>>
>>
>>
>> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
>>
>>
>>
>> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
>>
>>
>>
>> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
>>
>>
>>
>> 2018-10-12 08:21:35,838 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
>>
>> 2018-10-12 08:21:35,838 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
>>
>> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>>
>> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@17900f5b Connection State changed to SUSPENDED
>>
>>
>>
>> From: Mike Thomsen <mi...@gmail.com>
>> Sent: Friday, October 12, 2018 8:33 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
>>
>>
>>
>> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com> wrote:
>>
>> Alexander,
>>
>>
>>
>> I am pretty sure your problem is here: nifi.state.management.embedded.zookeeper.start=true
>>
>>
>>
>> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
>>
>>
>>
>> What you need to try is these steps:
>>
>>
>>
>> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be odd numbers)
>>
>> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>>
>> 3. Restart all of them.
>>
>>
>>
>> See if that works.
>>
>>
>>
>> Mike
>>
>>
>>
>> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
>>
>>
>>
>> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
>>
>> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
>>
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>
>> at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
>>
>> at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
>>
>> at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
>>
>> at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
>>
>> at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>> Is there anything else we should check?
>>
>>
>>
>> From: Nathan Gough <th...@gmail.com>
>> Sent: Thursday, October 11, 2018 9:12 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
>>
>>
>>
>> From: ashmeet kandhari <as...@gmail.com>
>> Reply-To: <us...@nifi.apache.org>
>> Date: Thursday, October 11, 2018 at 9:09 AM
>> To: <us...@nifi.apache.org>
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> Hi Alexander,
>>
>>
>>
>> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
>>
>>
>>
>> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
>>
>>
>>
>> From: ashmeet kandhari <as...@gmail.com>
>> Sent: Thursday, October 11, 2018 4:36 AM
>> To: users@nifi.apache.org
>> Subject: Re: NiFi fails on cluster nodes
>>
>>
>>
>> Can you see if zookeeper node is up and running and can connect to the nifi nodes
>>
>>
>>
>> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>>
>> Hello,
>>
>>
>>
>> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
>>
>>
>>
>> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>>
>> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>>
>> 2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
>>
>> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
>>
>> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
>>
>> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
>>
>> java.net.ConnectException: Connection timed out (Connection timed out)
>>
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>
>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>
>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>
>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>
>> at java.net.Socket.connect(Socket.java:589)
>>
>> at java.net.Socket.connect(Socket.java:538)
>>
>> at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
>>
>> at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>>
>> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>>
>> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>>
>> at org.apache.nifi.NiFi.main(NiFi.java:292)
>>
>> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
>>
>> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
>>
>>
>>
>> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
>>
>>
>>
>> Thanks in advance,
>>
>>
>>
>> Alexander
Re: NiFi fails on cluster nodes
Posted by Mike Thomsen <mi...@gmail.com>.
Not sure what's going on here, but NiFi does not require a cert to setup
ZooKeeper.
Mike
On Mon, Oct 15, 2018 at 9:39 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
alexander.saip@nih.gov> wrote:
> Hi Mike and Bryan,
>
>
>
> I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi
> node so far. Here is the error from the NiFi log:
>
>
>
> 2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol Request-1]
> o.a.nifi.security.util.CertificateUtils The incoming request did not
> contain client certificates and thus the DN cannot be extracted. Check that
> the other endpoint is providing a complete client certificate chain
>
> 2018-10-15 09:19:48,425 INFO [main]
> o.a.nifi.controller.StandardFlowService Connecting Node: 0.0.0.0:8008
>
> 2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol Request-2]
> o.a.nifi.security.util.CertificateUtils The incoming request did not
> contain client certificates and thus the DN cannot be extracted. Check that
> the other endpoint is providing a complete client certificate chain
>
> 2018-10-15 09:19:48,456 WARN [main]
> o.a.nifi.controller.StandardFlowService Failed to connect to cluster due
> to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling
> 'CONNECTION_REQUEST' protocol message due to:
> javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
>
>
>
> It is likely extraneous to NiFi, but does this mean that we need install a
> cert into ZooKeeper? Right now, both apps are running on the same box.
>
>
>
> Thank you.
>
>
>
> *From:* Mike Thomsen <mi...@gmail.com>
> *Sent:* Monday, October 15, 2018 9:02 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
>
>
>
> See the properties that start with "nifi.zookeeper."
>
>
>
> On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> Mike,
>
>
>
> I wonder if you could point me to instructions how to configure a cluster
> with an external instance of ZooKeeper? The NiFi Admin Guide talks
> exclusively about the embedded one.
>
>
>
> Thanks again.
>
>
>
> *From:* Mike Thomsen <mi...@gmail.com>
> *Sent:* Friday, October 12, 2018 10:17 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> It very well could become a problem down the road. The reason ZooKeeper is
> usually on a dedicated machine is that you want it to be able to have
> enough resources to always communicate within a quorum to reconcile
> configuration changes and feed configuration details to clients.
>
>
>
> That particular message is just a warning message. From what I can tell,
> it's just telling you that no cluster coordinator has been elected and it's
> going to try to do something about that. It's usually a problem with
> embedded ZooKeeper because each node by default points to the version of
> ZooKeeper it fires up.
>
>
>
> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores
> should be enough to run an external ZooKeeper.
>
>
>
> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess
> co-locating it with one of the NiFi nodes shouldn’t be an issue, or will
> it? We are chronically short of hardware. BTW, does the following message
> in the logs point to some sort of problem with the embedded ZooKeeper?
>
>
>
> 2018-10-12 08:21:35,838 WARN [main]
> o.a.nifi.controller.StandardFlowService There is currently no Cluster
> Coordinator. This often happens upon restart of NiFi when running an
> embedded ZooKeeper. Will register this node to become the active Cluster
> Coordinator and will attempt to connect to cluster again
>
> 2018-10-12 08:21:35,838 INFO [main]
> o.a.n.c.l.e.CuratorLeaderElectionManager
> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
> Election for role 'Cluster Coordinator' but this role is already registered
>
> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>
> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
> o.a.n.c.l.e.CuratorLeaderElectionManager
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@17900f5b
> Connection State changed to SUSPENDED
>
>
>
> *From:* Mike Thomsen <mi...@gmail.com>
> *Sent:* Friday, October 12, 2018 8:33 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> Also, in a production environment NiFi should have its own dedicated
> ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper
> quora (ex. have HBase and NiFi point to the same quorum).
>
>
>
> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>
> wrote:
>
> Alexander,
>
>
>
> I am pretty sure your problem is here:
> *nifi.state.management.embedded.zookeeper.start=true*
>
>
>
> That spins up an embedded ZooKeeper, which is generally intended to be
> used for local development. For example, HBase provides the same feature,
> but it is intended to allow you to test a real HBase client application
> against a single node of HBase running locally.
>
>
>
> What you need to try is these steps:
>
>
>
> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be
> odd numbers)
>
> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>
> 3. Restart all of them.
>
>
>
> See if that works.
>
>
>
> Mike
>
>
>
> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> *nifi.cluster.node.protocol.port=11443* by default on all nodes, I
> haven’t touched that property. Yesterday, we discovered some issues
> preventing two of the boxes from communicating. Now, they can talk okay.
> Ports 11443, 2181 and 3888 are explicitly open in *iptables*, but
> clustering still doesn’t happen. The log files are filled up with errors
> like this:
>
>
>
> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Is there anything else we should check?
>
>
>
> *From:* Nathan Gough <th...@gmail.com>
> *Sent:* Thursday, October 11, 2018 9:12 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on
> all nodes to allow cluster communication for cluster heartbeats etc.
>
>
>
> *From: *ashmeet kandhari <as...@gmail.com>
> *Reply-To: *<us...@nifi.apache.org>
> *Date: *Thursday, October 11, 2018 at 9:09 AM
> *To: *<us...@nifi.apache.org>
> *Subject: *Re: NiFi fails on cluster nodes
>
>
>
> Hi Alexander,
>
>
>
> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in
> standalone mode and see if you can ping them from other 2 servers just to
> be sure if they can communicate with one another.
>
>
>
> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> How do I do that? The *nifi.properties* file on each node includes ‘
> *nifi.state.management.embedded.zookeeper.start=true’*, so I assume
> Zookeeper does start.
>
>
>
> *From:* ashmeet kandhari <as...@gmail.com>
> *Sent:* Thursday, October 11, 2018 4:36 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> Can you see if zookeeper node is up and running and can connect to the
> nifi nodes
>
>
>
> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> Hello,
>
>
>
> We have three NiFi 1.7.1 nodes originally configured as independent
> instances, each on its own server. There is no firewall between them. When
> I tried to build a cluster following instructions here
> <https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>,
> NiFi failed to start on all of them, despite the fact that I even set *
> nifi.cluster.protocol.is.secure=false* in the *nifi.properties* file on
> each node. Here is the error in the log files:
>
>
>
> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> 2018-10-10 13:57:07,745 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties
> path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> 2018-10-10 13:57:07,748 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
> /opt/nifi-1.7.1/./conf/nifi.properties
>
> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
> properties
>
> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener
> Started Bootstrap Listener, Listening for incoming requests on port 43744
>
> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.net.ConnectException: Connection timed out
> (Connection timed out)
>
> java.net.ConnectException: Connection timed out (Connection timed out)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at java.net.Socket.connect(Socket.java:538)
>
> at
> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
>
> at
> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating
> shutdown of Jetty web server...
>
> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web
> server shutdown completed (nicely or otherwise).
>
>
>
> Without clustering, the instances had no problem starting. Since this is
> our first experiment building a cluster, I’m not sure where to look for
> clues.
>
>
>
> Thanks in advance,
>
>
>
> Alexander
>
>
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Hi Mike and Bryan,
I’ve installed and started ZooKeeper 3.4.13 and re-started a single NiFi node so far. Here is the error from the NiFi log:
2018-10-15 09:19:48,371 ERROR [Process Cluster Protocol Request-1] o.a.nifi.security.util.CertificateUtils The incoming request did not contain client certificates and thus the DN cannot be extracted. Check that the other endpoint is providing a complete client certificate chain
2018-10-15 09:19:48,425 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: 0.0.0.0:8008
2018-10-15 09:19:48,452 ERROR [Process Cluster Protocol Request-2] o.a.nifi.security.util.CertificateUtils The incoming request did not contain client certificates and thus the DN cannot be extracted. Check that the other endpoint is providing a complete client certificate chain
2018-10-15 09:19:48,456 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
It is likely extraneous to NiFi, but does this mean that we need install a cert into ZooKeeper? Right now, both apps are running on the same box.
Thank you.
From: Mike Thomsen <mi...@gmail.com>
Sent: Monday, October 15, 2018 9:02 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
See the properties that start with "nifi.zookeeper."
On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
Mike,
I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
Thanks again.
From: Mike Thomsen <mi...@gmail.com>>
Sent: Friday, October 12, 2018 10:17 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
2018-10-12 08:21:35,838 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2018-10-12 08:21:35,838 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
2018-10-12 08:21:42,090 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@17900f5b<ma...@17900f5b> Connection State changed to SUSPENDED
From: Mike Thomsen <mi...@gmail.com>>
Sent: Friday, October 12, 2018 8:33 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>> wrote:
Alexander,
I am pretty sure your problem is here: nifi.state.management.embedded.zookeeper.start=true
That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
What you need to try is these steps:
1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be odd numbers)
2. Update nifi.properties on each node to use the external ZooKeeper setup.
3. Restart all of them.
See if that works.
Mike
On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
2018-10-12 07:59:08,494 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is there anything else we should check?
From: Nathan Gough <th...@gmail.com>>
Sent: Thursday, October 11, 2018 9:12 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
From: ashmeet kandhari <as...@gmail.com>>
Reply-To: <us...@nifi.apache.org>>
Date: Thursday, October 11, 2018 at 9:09 AM
To: <us...@nifi.apache.org>>
Subject: Re: NiFi fails on cluster nodes
Hi Alexander,
Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
From: ashmeet kandhari <as...@gmail.com>>
Sent: Thursday, October 11, 2018 4:36 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
Can you see if zookeeper node is up and running and can connect to the nifi nodes
On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
Hello,
We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here<https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
at org.apache.nifi.NiFi.<init>(NiFi.java:102)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
Thanks in advance,
Alexander
Re: NiFi fails on cluster nodes
Posted by Mike Thomsen <mi...@gmail.com>.
http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
See the properties that start with "nifi.zookeeper."
On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
alexander.saip@nih.gov> wrote:
> Mike,
>
>
>
> I wonder if you could point me to instructions how to configure a cluster
> with an external instance of ZooKeeper? The NiFi Admin Guide talks
> exclusively about the embedded one.
>
>
>
> Thanks again.
>
>
>
> *From:* Mike Thomsen <mi...@gmail.com>
> *Sent:* Friday, October 12, 2018 10:17 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> It very well could become a problem down the road. The reason ZooKeeper is
> usually on a dedicated machine is that you want it to be able to have
> enough resources to always communicate within a quorum to reconcile
> configuration changes and feed configuration details to clients.
>
>
>
> That particular message is just a warning message. From what I can tell,
> it's just telling you that no cluster coordinator has been elected and it's
> going to try to do something about that. It's usually a problem with
> embedded ZooKeeper because each node by default points to the version of
> ZooKeeper it fires up.
>
>
>
> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores
> should be enough to run an external ZooKeeper.
>
>
>
> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess
> co-locating it with one of the NiFi nodes shouldn’t be an issue, or will
> it? We are chronically short of hardware. BTW, does the following message
> in the logs point to some sort of problem with the embedded ZooKeeper?
>
>
>
> 2018-10-12 08:21:35,838 WARN [main]
> o.a.nifi.controller.StandardFlowService There is currently no Cluster
> Coordinator. This often happens upon restart of NiFi when running an
> embedded ZooKeeper. Will register this node to become the active Cluster
> Coordinator and will attempt to connect to cluster again
>
> 2018-10-12 08:21:35,838 INFO [main]
> o.a.n.c.l.e.CuratorLeaderElectionManager
> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
> Election for role 'Cluster Coordinator' but this role is already registered
>
> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>
> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
> o.a.n.c.l.e.CuratorLeaderElectionManager
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@17900f5b
> Connection State changed to SUSPENDED
>
>
>
> *From:* Mike Thomsen <mi...@gmail.com>
> *Sent:* Friday, October 12, 2018 8:33 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> Also, in a production environment NiFi should have its own dedicated
> ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper
> quora (ex. have HBase and NiFi point to the same quorum).
>
>
>
> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>
> wrote:
>
> Alexander,
>
>
>
> I am pretty sure your problem is here:
> *nifi.state.management.embedded.zookeeper.start=true*
>
>
>
> That spins up an embedded ZooKeeper, which is generally intended to be
> used for local development. For example, HBase provides the same feature,
> but it is intended to allow you to test a real HBase client application
> against a single node of HBase running locally.
>
>
>
> What you need to try is these steps:
>
>
>
> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be
> odd numbers)
>
> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>
> 3. Restart all of them.
>
>
>
> See if that works.
>
>
>
> Mike
>
>
>
> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> *nifi.cluster.node.protocol.port=11443* by default on all nodes, I
> haven’t touched that property. Yesterday, we discovered some issues
> preventing two of the boxes from communicating. Now, they can talk okay.
> Ports 11443, 2181 and 3888 are explicitly open in *iptables*, but
> clustering still doesn’t happen. The log files are filled up with errors
> like this:
>
>
>
> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Is there anything else we should check?
>
>
>
> *From:* Nathan Gough <th...@gmail.com>
> *Sent:* Thursday, October 11, 2018 9:12 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on
> all nodes to allow cluster communication for cluster heartbeats etc.
>
>
>
> *From: *ashmeet kandhari <as...@gmail.com>
> *Reply-To: *<us...@nifi.apache.org>
> *Date: *Thursday, October 11, 2018 at 9:09 AM
> *To: *<us...@nifi.apache.org>
> *Subject: *Re: NiFi fails on cluster nodes
>
>
>
> Hi Alexander,
>
>
>
> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in
> standalone mode and see if you can ping them from other 2 servers just to
> be sure if they can communicate with one another.
>
>
>
> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> How do I do that? The *nifi.properties* file on each node includes ‘
> *nifi.state.management.embedded.zookeeper.start=true’*, so I assume
> Zookeeper does start.
>
>
>
> *From:* ashmeet kandhari <as...@gmail.com>
> *Sent:* Thursday, October 11, 2018 4:36 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> Can you see if zookeeper node is up and running and can connect to the
> nifi nodes
>
>
>
> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> Hello,
>
>
>
> We have three NiFi 1.7.1 nodes originally configured as independent
> instances, each on its own server. There is no firewall between them. When
> I tried to build a cluster following instructions here
> <https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>,
> NiFi failed to start on all of them, despite the fact that I even set *
> nifi.cluster.protocol.is.secure=false* in the *nifi.properties* file on
> each node. Here is the error in the log files:
>
>
>
> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> 2018-10-10 13:57:07,745 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties
> path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> 2018-10-10 13:57:07,748 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
> /opt/nifi-1.7.1/./conf/nifi.properties
>
> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
> properties
>
> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener
> Started Bootstrap Listener, Listening for incoming requests on port 43744
>
> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.net.ConnectException: Connection timed out
> (Connection timed out)
>
> java.net.ConnectException: Connection timed out (Connection timed out)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at java.net.Socket.connect(Socket.java:538)
>
> at
> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
>
> at
> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating
> shutdown of Jetty web server...
>
> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web
> server shutdown completed (nicely or otherwise).
>
>
>
> Without clustering, the instances had no problem starting. Since this is
> our first experiment building a cluster, I’m not sure where to look for
> clues.
>
>
>
> Thanks in advance,
>
>
>
> Alexander
>
>
Re: NiFi fails on cluster nodes
Posted by Bryan Bende <bb...@gmail.com>.
The cluster configuration section of the admin guide [1] is
independent of whether it is embedded or external zookeeper.
The only real difference is you won't set
nifi.state.management.embedded.zookeeper.start=true, but besides that
you all of the other config would be the same whether using embedded
or external.
[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
On Mon, Oct 15, 2018 at 8:58 AM Saip, Alexander (NIH/CC/BTRIS) [C]
<al...@nih.gov> wrote:
>
> Mike,
>
>
>
> I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
>
>
>
> Thanks again.
>
>
>
> From: Mike Thomsen <mi...@gmail.com>
> Sent: Friday, October 12, 2018 10:17 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
>
>
>
> That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
>
>
>
> For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
>
>
>
> On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
>
>
>
> 2018-10-12 08:21:35,838 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
>
> 2018-10-12 08:21:35,838 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
>
> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>
> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@17900f5b Connection State changed to SUSPENDED
>
>
>
> From: Mike Thomsen <mi...@gmail.com>
> Sent: Friday, October 12, 2018 8:33 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
>
>
>
> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> Alexander,
>
>
>
> I am pretty sure your problem is here: nifi.state.management.embedded.zookeeper.start=true
>
>
>
> That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
>
>
>
> What you need to try is these steps:
>
>
>
> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be odd numbers)
>
> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>
> 3. Restart all of them.
>
>
>
> See if that works.
>
>
>
> Mike
>
>
>
> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
>
>
>
> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
>
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
>
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
>
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
>
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
>
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
>
> at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Is there anything else we should check?
>
>
>
> From: Nathan Gough <th...@gmail.com>
> Sent: Thursday, October 11, 2018 9:12 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
>
>
>
> From: ashmeet kandhari <as...@gmail.com>
> Reply-To: <us...@nifi.apache.org>
> Date: Thursday, October 11, 2018 at 9:09 AM
> To: <us...@nifi.apache.org>
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Hi Alexander,
>
>
>
> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
>
>
>
> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
>
>
>
> From: ashmeet kandhari <as...@gmail.com>
> Sent: Thursday, October 11, 2018 4:36 AM
> To: users@nifi.apache.org
> Subject: Re: NiFi fails on cluster nodes
>
>
>
> Can you see if zookeeper node is up and running and can connect to the nifi nodes
>
>
>
> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
>
> Hello,
>
>
>
> We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
>
>
>
> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> 2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> 2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
>
> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
>
> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
>
> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
>
> java.net.ConnectException: Connection timed out (Connection timed out)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at java.net.Socket.connect(Socket.java:538)
>
> at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
>
> at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
>
> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
>
>
>
> Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
>
>
>
> Thanks in advance,
>
>
>
> Alexander
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Mike,
I wonder if you could point me to instructions how to configure a cluster with an external instance of ZooKeeper? The NiFi Admin Guide talks exclusively about the embedded one.
Thanks again.
From: Mike Thomsen <mi...@gmail.com>
Sent: Friday, October 12, 2018 10:17 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
It very well could become a problem down the road. The reason ZooKeeper is usually on a dedicated machine is that you want it to be able to have enough resources to always communicate within a quorum to reconcile configuration changes and feed configuration details to clients.
That particular message is just a warning message. From what I can tell, it's just telling you that no cluster coordinator has been elected and it's going to try to do something about that. It's usually a problem with embedded ZooKeeper because each node by default points to the version of ZooKeeper it fires up.
For a development environment, a VM with 2GB of RAM and 1-2 CPU cores should be enough to run an external ZooKeeper.
On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
2018-10-12 08:21:35,838 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2018-10-12 08:21:35,838 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
2018-10-12 08:21:42,090 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@17900f5b<ma...@17900f5b> Connection State changed to SUSPENDED
From: Mike Thomsen <mi...@gmail.com>>
Sent: Friday, October 12, 2018 8:33 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>> wrote:
Alexander,
I am pretty sure your problem is here: nifi.state.management.embedded.zookeeper.start=true
That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
What you need to try is these steps:
1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be odd numbers)
2. Update nifi.properties on each node to use the external ZooKeeper setup.
3. Restart all of them.
See if that works.
Mike
On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
2018-10-12 07:59:08,494 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is there anything else we should check?
From: Nathan Gough <th...@gmail.com>>
Sent: Thursday, October 11, 2018 9:12 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
From: ashmeet kandhari <as...@gmail.com>>
Reply-To: <us...@nifi.apache.org>>
Date: Thursday, October 11, 2018 at 9:09 AM
To: <us...@nifi.apache.org>>
Subject: Re: NiFi fails on cluster nodes
Hi Alexander,
Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
From: ashmeet kandhari <as...@gmail.com>>
Sent: Thursday, October 11, 2018 4:36 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
Can you see if zookeeper node is up and running and can connect to the nifi nodes
On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
Hello,
We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here<https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
at org.apache.nifi.NiFi.<init>(NiFi.java:102)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
Thanks in advance,
Alexander
Re: NiFi fails on cluster nodes
Posted by Mike Thomsen <mi...@gmail.com>.
It very well could become a problem down the road. The reason ZooKeeper is
usually on a dedicated machine is that you want it to be able to have
enough resources to always communicate within a quorum to reconcile
configuration changes and feed configuration details to clients.
That particular message is just a warning message. From what I can tell,
it's just telling you that no cluster coordinator has been elected and it's
going to try to do something about that. It's usually a problem with
embedded ZooKeeper because each node by default points to the version of
ZooKeeper it fires up.
For a development environment, a VM with 2GB of RAM and 1-2 CPU cores
should be enough to run an external ZooKeeper.
On Fri, Oct 12, 2018 at 9:47 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
alexander.saip@nih.gov> wrote:
> Thanks Mike. We will get an external ZooKeeper instance deployed. I guess
> co-locating it with one of the NiFi nodes shouldn’t be an issue, or will
> it? We are chronically short of hardware. BTW, does the following message
> in the logs point to some sort of problem with the embedded ZooKeeper?
>
>
>
> 2018-10-12 08:21:35,838 WARN [main]
> o.a.nifi.controller.StandardFlowService There is currently no Cluster
> Coordinator. This often happens upon restart of NiFi when running an
> embedded ZooKeeper. Will register this node to become the active Cluster
> Coordinator and will attempt to connect to cluster again
>
> 2018-10-12 08:21:35,838 INFO [main]
> o.a.n.c.l.e.CuratorLeaderElectionManager
> CuratorLeaderElectionManager[stopped=false] Attempted to register Leader
> Election for role 'Cluster Coordinator' but this role is already registered
>
> 2018-10-12 08:21:42,090 INFO [Curator-Framework-0]
> o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
>
> 2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0]
> o.a.n.c.l.e.CuratorLeaderElectionManager
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@17900f5b
> Connection State changed to SUSPENDED
>
>
>
> *From:* Mike Thomsen <mi...@gmail.com>
> *Sent:* Friday, October 12, 2018 8:33 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> Also, in a production environment NiFi should have its own dedicated
> ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper
> quora (ex. have HBase and NiFi point to the same quorum).
>
>
>
> On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>
> wrote:
>
> Alexander,
>
>
>
> I am pretty sure your problem is here:
> *nifi.state.management.embedded.zookeeper.start=true*
>
>
>
> That spins up an embedded ZooKeeper, which is generally intended to be
> used for local development. For example, HBase provides the same feature,
> but it is intended to allow you to test a real HBase client application
> against a single node of HBase running locally.
>
>
>
> What you need to try is these steps:
>
>
>
> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be
> odd numbers)
>
> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
>
> 3. Restart all of them.
>
>
>
> See if that works.
>
>
>
> Mike
>
>
>
> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> *nifi.cluster.node.protocol.port=11443* by default on all nodes, I
> haven’t touched that property. Yesterday, we discovered some issues
> preventing two of the boxes from communicating. Now, they can talk okay.
> Ports 11443, 2181 and 3888 are explicitly open in *iptables*, but
> clustering still doesn’t happen. The log files are filled up with errors
> like this:
>
>
>
> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Is there anything else we should check?
>
>
>
> *From:* Nathan Gough <th...@gmail.com>
> *Sent:* Thursday, October 11, 2018 9:12 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on
> all nodes to allow cluster communication for cluster heartbeats etc.
>
>
>
> *From: *ashmeet kandhari <as...@gmail.com>
> *Reply-To: *<us...@nifi.apache.org>
> *Date: *Thursday, October 11, 2018 at 9:09 AM
> *To: *<us...@nifi.apache.org>
> *Subject: *Re: NiFi fails on cluster nodes
>
>
>
> Hi Alexander,
>
>
>
> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in
> standalone mode and see if you can ping them from other 2 servers just to
> be sure if they can communicate with one another.
>
>
>
> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> How do I do that? The *nifi.properties* file on each node includes ‘
> *nifi.state.management.embedded.zookeeper.start=true’*, so I assume
> Zookeeper does start.
>
>
>
> *From:* ashmeet kandhari <as...@gmail.com>
> *Sent:* Thursday, October 11, 2018 4:36 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> Can you see if zookeeper node is up and running and can connect to the
> nifi nodes
>
>
>
> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> Hello,
>
>
>
> We have three NiFi 1.7.1 nodes originally configured as independent
> instances, each on its own server. There is no firewall between them. When
> I tried to build a cluster following instructions here
> <https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>,
> NiFi failed to start on all of them, despite the fact that I even set *
> nifi.cluster.protocol.is.secure=false* in the *nifi.properties* file on
> each node. Here is the error in the log files:
>
>
>
> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> 2018-10-10 13:57:07,745 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties
> path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> 2018-10-10 13:57:07,748 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
> /opt/nifi-1.7.1/./conf/nifi.properties
>
> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
> properties
>
> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener
> Started Bootstrap Listener, Listening for incoming requests on port 43744
>
> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.net.ConnectException: Connection timed out
> (Connection timed out)
>
> java.net.ConnectException: Connection timed out (Connection timed out)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at java.net.Socket.connect(Socket.java:538)
>
> at
> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
>
> at
> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating
> shutdown of Jetty web server...
>
> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web
> server shutdown completed (nicely or otherwise).
>
>
>
> Without clustering, the instances had no problem starting. Since this is
> our first experiment building a cluster, I’m not sure where to look for
> clues.
>
>
>
> Thanks in advance,
>
>
>
> Alexander
>
>
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
Thanks Mike. We will get an external ZooKeeper instance deployed. I guess co-locating it with one of the NiFi nodes shouldn’t be an issue, or will it? We are chronically short of hardware. BTW, does the following message in the logs point to some sort of problem with the embedded ZooKeeper?
2018-10-12 08:21:35,838 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2018-10-12 08:21:35,838 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
2018-10-12 08:21:42,090 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2018-10-12 08:21:42,092 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@17900f5b Connection State changed to SUSPENDED
From: Mike Thomsen <mi...@gmail.com>
Sent: Friday, October 12, 2018 8:33 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
Also, in a production environment NiFi should have its own dedicated ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper quora (ex. have HBase and NiFi point to the same quorum).
On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com>> wrote:
Alexander,
I am pretty sure your problem is here: nifi.state.management.embedded.zookeeper.start=true
That spins up an embedded ZooKeeper, which is generally intended to be used for local development. For example, HBase provides the same feature, but it is intended to allow you to test a real HBase client application against a single node of HBase running locally.
What you need to try is these steps:
1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be odd numbers)
2. Update nifi.properties on each node to use the external ZooKeeper setup.
3. Restart all of them.
See if that works.
Mike
On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
2018-10-12 07:59:08,494 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is there anything else we should check?
From: Nathan Gough <th...@gmail.com>>
Sent: Thursday, October 11, 2018 9:12 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
From: ashmeet kandhari <as...@gmail.com>>
Reply-To: <us...@nifi.apache.org>>
Date: Thursday, October 11, 2018 at 9:09 AM
To: <us...@nifi.apache.org>>
Subject: Re: NiFi fails on cluster nodes
Hi Alexander,
Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
From: ashmeet kandhari <as...@gmail.com>>
Sent: Thursday, October 11, 2018 4:36 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
Can you see if zookeeper node is up and running and can connect to the nifi nodes
On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
Hello,
We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here<https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
at org.apache.nifi.NiFi.<init>(NiFi.java:102)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
Thanks in advance,
Alexander
Re: NiFi fails on cluster nodes
Posted by Mike Thomsen <mi...@gmail.com>.
Also, in a production environment NiFi should have its own dedicated
ZooKeeper cluster to be on the safe side. You should not reuse ZooKeeper
quora (ex. have HBase and NiFi point to the same quorum).
On Fri, Oct 12, 2018 at 8:29 AM Mike Thomsen <mi...@gmail.com> wrote:
> Alexander,
>
> I am pretty sure your problem is here:
> *nifi.state.management.embedded.zookeeper.start=true*
>
> That spins up an embedded ZooKeeper, which is generally intended to be
> used for local development. For example, HBase provides the same feature,
> but it is intended to allow you to test a real HBase client application
> against a single node of HBase running locally.
>
> What you need to try is these steps:
>
> 1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be
> odd numbers)
> 2. Update nifi.properties on each node to use the external ZooKeeper setup.
> 3. Restart all of them.
>
> See if that works.
>
> Mike
>
> On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
>> *nifi.cluster.node.protocol.port=11443* by default on all nodes, I
>> haven’t touched that property. Yesterday, we discovered some issues
>> preventing two of the boxes from communicating. Now, they can talk okay.
>> Ports 11443, 2181 and 3888 are explicitly open in *iptables*, but
>> clustering still doesn’t happen. The log files are filled up with errors
>> like this:
>>
>>
>>
>> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
>> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
>>
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss
>>
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
>>
>> at
>> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>> Is there anything else we should check?
>>
>>
>>
>> *From:* Nathan Gough <th...@gmail.com>
>> *Sent:* Thursday, October 11, 2018 9:12 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: NiFi fails on cluster nodes
>>
>>
>>
>> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on
>> all nodes to allow cluster communication for cluster heartbeats etc.
>>
>>
>>
>> *From: *ashmeet kandhari <as...@gmail.com>
>> *Reply-To: *<us...@nifi.apache.org>
>> *Date: *Thursday, October 11, 2018 at 9:09 AM
>> *To: *<us...@nifi.apache.org>
>> *Subject: *Re: NiFi fails on cluster nodes
>>
>>
>>
>> Hi Alexander,
>>
>>
>>
>> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in
>> standalone mode and see if you can ping them from other 2 servers just to
>> be sure if they can communicate with one another.
>>
>>
>>
>> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
>> alexander.saip@nih.gov> wrote:
>>
>> How do I do that? The *nifi.properties* file on each node includes ‘
>> *nifi.state.management.embedded.zookeeper.start=true’*, so I assume
>> Zookeeper does start.
>>
>>
>>
>> *From:* ashmeet kandhari <as...@gmail.com>
>> *Sent:* Thursday, October 11, 2018 4:36 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: NiFi fails on cluster nodes
>>
>>
>>
>> Can you see if zookeeper node is up and running and can connect to the
>> nifi nodes
>>
>>
>>
>> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <
>> alexander.saip@nih.gov> wrote:
>>
>> Hello,
>>
>>
>>
>> We have three NiFi 1.7.1 nodes originally configured as independent
>> instances, each on its own server. There is no firewall between them. When
>> I tried to build a cluster following instructions here
>> <https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>,
>> NiFi failed to start on all of them, despite the fact that I even set *
>> nifi.cluster.protocol.is.secure=false* in the *nifi.properties* file on
>> each node. Here is the error in the log files:
>>
>>
>>
>> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>>
>> 2018-10-10 13:57:07,745 INFO [main]
>> o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties
>> path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>>
>> 2018-10-10 13:57:07,748 INFO [main]
>> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
>> /opt/nifi-1.7.1/./conf/nifi.properties
>>
>> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
>> properties
>>
>> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener
>> Started Bootstrap Listener, Listening for incoming requests on port 43744
>>
>> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to
>> launch NiFi due to java.net.ConnectException: Connection timed out
>> (Connection timed out)
>>
>> java.net.ConnectException: Connection timed out (Connection timed out)
>>
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>
>> at java.net.Socket.connect(Socket.java:589)
>>
>> at java.net.Socket.connect(Socket.java:538)
>>
>> at
>> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
>>
>> at
>> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>>
>> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>>
>> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>>
>> at org.apache.nifi.NiFi.main(NiFi.java:292)
>>
>> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating
>> shutdown of Jetty web server...
>>
>> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web
>> server shutdown completed (nicely or otherwise).
>>
>>
>>
>> Without clustering, the instances had no problem starting. Since this is
>> our first experiment building a cluster, I’m not sure where to look for
>> clues.
>>
>>
>>
>> Thanks in advance,
>>
>>
>>
>> Alexander
>>
>>
Re: NiFi fails on cluster nodes
Posted by Mike Thomsen <mi...@gmail.com>.
Alexander,
I am pretty sure your problem is here:
*nifi.state.management.embedded.zookeeper.start=true*
That spins up an embedded ZooKeeper, which is generally intended to be used
for local development. For example, HBase provides the same feature, but it
is intended to allow you to test a real HBase client application against a
single node of HBase running locally.
What you need to try is these steps:
1. Set up an external ZooKeeper instance (or set up 3 in a quorum; must be
odd numbers)
2. Update nifi.properties on each node to use the external ZooKeeper setup.
3. Restart all of them.
See if that works.
Mike
On Fri, Oct 12, 2018 at 8:13 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
alexander.saip@nih.gov> wrote:
> *nifi.cluster.node.protocol.port=11443* by default on all nodes, I
> haven’t touched that property. Yesterday, we discovered some issues
> preventing two of the boxes from communicating. Now, they can talk okay.
> Ports 11443, 2181 and 3888 are explicitly open in *iptables*, but
> clustering still doesn’t happen. The log files are filled up with errors
> like this:
>
>
>
> 2018-10-12 07:59:08,494 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
>
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
>
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Is there anything else we should check?
>
>
>
> *From:* Nathan Gough <th...@gmail.com>
> *Sent:* Thursday, October 11, 2018 9:12 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on
> all nodes to allow cluster communication for cluster heartbeats etc.
>
>
>
> *From: *ashmeet kandhari <as...@gmail.com>
> *Reply-To: *<us...@nifi.apache.org>
> *Date: *Thursday, October 11, 2018 at 9:09 AM
> *To: *<us...@nifi.apache.org>
> *Subject: *Re: NiFi fails on cluster nodes
>
>
>
> Hi Alexander,
>
>
>
> Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in
> standalone mode and see if you can ping them from other 2 servers just to
> be sure if they can communicate with one another.
>
>
>
> On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> How do I do that? The *nifi.properties* file on each node includes ‘
> *nifi.state.management.embedded.zookeeper.start=true’*, so I assume
> Zookeeper does start.
>
>
>
> *From:* ashmeet kandhari <as...@gmail.com>
> *Sent:* Thursday, October 11, 2018 4:36 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> Can you see if zookeeper node is up and running and can connect to the
> nifi nodes
>
>
>
> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> Hello,
>
>
>
> We have three NiFi 1.7.1 nodes originally configured as independent
> instances, each on its own server. There is no firewall between them. When
> I tried to build a cluster following instructions here
> <https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>,
> NiFi failed to start on all of them, despite the fact that I even set *
> nifi.cluster.protocol.is.secure=false* in the *nifi.properties* file on
> each node. Here is the error in the log files:
>
>
>
> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> 2018-10-10 13:57:07,745 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties
> path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> 2018-10-10 13:57:07,748 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
> /opt/nifi-1.7.1/./conf/nifi.properties
>
> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
> properties
>
> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener
> Started Bootstrap Listener, Listening for incoming requests on port 43744
>
> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.net.ConnectException: Connection timed out
> (Connection timed out)
>
> java.net.ConnectException: Connection timed out (Connection timed out)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at java.net.Socket.connect(Socket.java:538)
>
> at
> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
>
> at
> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating
> shutdown of Jetty web server...
>
> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web
> server shutdown completed (nicely or otherwise).
>
>
>
> Without clustering, the instances had no problem starting. Since this is
> our first experiment building a cluster, I’m not sure where to look for
> clues.
>
>
>
> Thanks in advance,
>
>
>
> Alexander
>
>
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
nifi.cluster.node.protocol.port=11443 by default on all nodes, I haven’t touched that property. Yesterday, we discovered some issues preventing two of the boxes from communicating. Now, they can talk okay. Ports 11443, 2181 and 3888 are explicitly open in iptables, but clustering still doesn’t happen. The log files are filled up with errors like this:
2018-10-12 07:59:08,494 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is there anything else we should check?
From: Nathan Gough <th...@gmail.com>
Sent: Thursday, October 11, 2018 9:12 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
From: ashmeet kandhari <as...@gmail.com>>
Reply-To: <us...@nifi.apache.org>>
Date: Thursday, October 11, 2018 at 9:09 AM
To: <us...@nifi.apache.org>>
Subject: Re: NiFi fails on cluster nodes
Hi Alexander,
Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
From: ashmeet kandhari <as...@gmail.com>>
Sent: Thursday, October 11, 2018 4:36 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
Can you see if zookeeper node is up and running and can connect to the nifi nodes
On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
Hello,
We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here<https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
at org.apache.nifi.NiFi.<init>(NiFi.java:102)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
Thanks in advance,
Alexander
Re: NiFi fails on cluster nodes
Posted by Nathan Gough <th...@gmail.com>.
You may also need to explicitly open ‘nifi.cluster.node.protocol.port’ on all nodes to allow cluster communication for cluster heartbeats etc.
From: ashmeet kandhari <as...@gmail.com>
Reply-To: <us...@nifi.apache.org>
Date: Thursday, October 11, 2018 at 9:09 AM
To: <us...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
Hi Alexander,
Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in standalone mode and see if you can ping them from other 2 servers just to be sure if they can communicate with one another.
On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
From: ashmeet kandhari <as...@gmail.com>
Sent: Thursday, October 11, 2018 4:36 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
Can you see if zookeeper node is up and running and can connect to the nifi nodes
On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov> wrote:
Hello,
We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
at org.apache.nifi.NiFi.<init>(NiFi.java:102)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
Thanks in advance,
Alexander
Re: NiFi fails on cluster nodes
Posted by ashmeet kandhari <as...@gmail.com>.
Hi Alexander,
Can you verify by pinging if the 3 nodes (tcp ping) or run nifi in
standalone mode and see if you can ping them from other 2 servers just to
be sure if they can communicate with one another.
On Thu, Oct 11, 2018 at 11:49 AM Saip, Alexander (NIH/CC/BTRIS) [C] <
alexander.saip@nih.gov> wrote:
> How do I do that? The *nifi.properties* file on each node includes ‘
> *nifi.state.management.embedded.zookeeper.start=true’*, so I assume
> Zookeeper does start.
>
>
>
> *From:* ashmeet kandhari <as...@gmail.com>
> *Sent:* Thursday, October 11, 2018 4:36 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi fails on cluster nodes
>
>
>
> Can you see if zookeeper node is up and running and can connect to the
> nifi nodes
>
>
>
> On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <
> alexander.saip@nih.gov> wrote:
>
> Hello,
>
>
>
> We have three NiFi 1.7.1 nodes originally configured as independent
> instances, each on its own server. There is no firewall between them. When
> I tried to build a cluster following instructions here
> <https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>,
> NiFi failed to start on all of them, despite the fact that I even set *
> nifi.cluster.protocol.is.secure=false* in the *nifi.properties* file on
> each node. Here is the error in the log files:
>
>
>
> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> 2018-10-10 13:57:07,745 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties
> path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> 2018-10-10 13:57:07,748 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
> /opt/nifi-1.7.1/./conf/nifi.properties
>
> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
> properties
>
> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener
> Started Bootstrap Listener, Listening for incoming requests on port 43744
>
> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.net.ConnectException: Connection timed out
> (Connection timed out)
>
> java.net.ConnectException: Connection timed out (Connection timed out)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at java.net.Socket.connect(Socket.java:538)
>
> at
> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
>
> at
> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating
> shutdown of Jetty web server...
>
> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web
> server shutdown completed (nicely or otherwise).
>
>
>
> Without clustering, the instances had no problem starting. Since this is
> our first experiment building a cluster, I’m not sure where to look for
> clues.
>
>
>
> Thanks in advance,
>
>
>
> Alexander
>
>
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
After I explicitly opened ports 2181 and 3888 on all the nodes, the NiFi instances start and run, but apparently, there is still no communication between them. Here is what gets written over and over in the nifi-app.log files:
2018-10-11 08:16:53,074 INFO [main] o.a.nifi.groups.StandardProcessGroup Template[id=f8a45adb-e68f-46c5-b627-4c9805ba74e7] added to StandardProcessGroup[identifier=31f52f8c-015d-1000-05e9-6fe2f3320429]
2018-10-11 08:16:53,080 INFO [main] o.a.nifi.groups.StandardProcessGroup Template[id=63489abd-fb73-4d26-9814-48e40511d77d] added to StandardProcessGroup[identifier=31f52f8c-015d-1000-05e9-6fe2f3320429]
2018-10-11 08:16:53,162 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow
2018-10-11 08:16:53,512 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: 0.0.0.0:8008
2018-10-11 08:17:00,781 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again
2018-10-11 08:17:00,781 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered
2018-10-11 08:17:05,802 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED
2018-10-11 08:17:05,804 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@4240468b Connection State changed to SUSPENDED
2018-10-11 08:17:05,804 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is there anything else I missed?
From: Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>
Sent: Thursday, October 11, 2018 6:50 AM
To: users@nifi.apache.org
Subject: RE: NiFi fails on cluster nodes
How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
From: ashmeet kandhari <as...@gmail.com>>
Sent: Thursday, October 11, 2018 4:36 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi fails on cluster nodes
Can you see if zookeeper node is up and running and can connect to the nifi nodes
On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
Hello,
We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here<https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
at org.apache.nifi.NiFi.<init>(NiFi.java:102)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
Thanks in advance,
Alexander
RE: NiFi fails on cluster nodes
Posted by "Saip, Alexander (NIH/CC/BTRIS) [C]" <al...@nih.gov>.
How do I do that? The nifi.properties file on each node includes ‘nifi.state.management.embedded.zookeeper.start=true’, so I assume Zookeeper does start.
From: ashmeet kandhari <as...@gmail.com>
Sent: Thursday, October 11, 2018 4:36 AM
To: users@nifi.apache.org
Subject: Re: NiFi fails on cluster nodes
Can you see if zookeeper node is up and running and can connect to the nifi nodes
On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <al...@nih.gov>> wrote:
Hello,
We have three NiFi 1.7.1 nodes originally configured as independent instances, each on its own server. There is no firewall between them. When I tried to build a cluster following instructions here<https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>, NiFi failed to start on all of them, despite the fact that I even set nifi.cluster.protocol.is.secure=false in the nifi.properties file on each node. Here is the error in the log files:
2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-10 13:57:07,745 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
2018-10-10 13:57:07,748 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from /opt/nifi-1.7.1/./conf/nifi.properties
2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125 properties
2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 43744
2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.net.ConnectException: Connection timed out (Connection timed out)
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
at org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
at org.apache.nifi.NiFi.<init>(NiFi.java:102)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:292)
2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Without clustering, the instances had no problem starting. Since this is our first experiment building a cluster, I’m not sure where to look for clues.
Thanks in advance,
Alexander
Re: NiFi fails on cluster nodes
Posted by ashmeet kandhari <as...@gmail.com>.
Can you see if zookeeper node is up and running and can connect to the nifi
nodes
On Wed, Oct 10, 2018 at 7:34 PM Saip, Alexander (NIH/CC/BTRIS) [C] <
alexander.saip@nih.gov> wrote:
> Hello,
>
>
>
> We have three NiFi 1.7.1 nodes originally configured as independent
> instances, each on its own server. There is no firewall between them. When
> I tried to build a cluster following instructions here
> <https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/>,
> NiFi failed to start on all of them, despite the fact that I even set
> *nifi.cluster.protocol.is.secure=false* in the *nifi.properties* file on
> each node. Here is the error in the log files:
>
>
>
> 2018-10-10 13:57:07,506 INFO [main] org.apache.nifi.NiFi Launching NiFi...
>
> 2018-10-10 13:57:07,745 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties
> path to be '/opt/nifi-1.7.1/./conf/nifi.properties'
>
> 2018-10-10 13:57:07,748 INFO [main]
> o.a.nifi.properties.NiFiPropertiesLoader Loaded 125 properties from
> /opt/nifi-1.7.1/./conf/nifi.properties
>
> 2018-10-10 13:57:07,755 INFO [main] org.apache.nifi.NiFi Loaded 125
> properties
>
> 2018-10-10 13:57:07,762 INFO [main] org.apache.nifi.BootstrapListener
> Started Bootstrap Listener, Listening for incoming requests on port 43744
>
> 2018-10-10 13:59:15,056 ERROR [main] org.apache.nifi.NiFi Failure to
> launch NiFi due to java.net.ConnectException: Connection timed out
> (Connection timed out)
>
> java.net.ConnectException: Connection timed out (Connection timed out)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at java.net.Socket.connect(Socket.java:538)
>
> at
> org.apache.nifi.BootstrapListener.sendCommand(BootstrapListener.java:100)
>
> at
> org.apache.nifi.BootstrapListener.start(BootstrapListener.java:83)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:102)
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:71)
>
> at org.apache.nifi.NiFi.main(NiFi.java:292)
>
> 2018-10-10 13:59:15,058 INFO [Thread-1] org.apache.nifi.NiFi Initiating
> shutdown of Jetty web server...
>
> 2018-10-10 13:59:15,059 INFO [Thread-1] org.apache.nifi.NiFi Jetty web
> server shutdown completed (nicely or otherwise).
>
>
>
> Without clustering, the instances had no problem starting. Since this is
> our first experiment building a cluster, I’m not sure where to look for
> clues.
>
>
>
> Thanks in advance,
>
>
>
> Alexander
>