You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Conrad Crampton <co...@SecData.com> on 2016/10/19 12:38:55 UTC
Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Hi,
I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
6 nodes with a NCM.
With the removal of NCM in 1.0.0 I believe I now have an issue where none of my Remote Process Groups work as they previously did because they were configured to connect to the NCM (as the RPG url) which now doesn’t exist.
I have tried converting my NCM to a node but whilst I can get it running (sort of) when I try and connect to the cluster I get something like this in my logs…
2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) [nifi-jetty-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.<init>(NiFi.java:152) [nifi-runtime-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.main(NiFi.java:243) [nifi-runtime-1.0.0.jar:1.0.0]
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857) ~[nifi-framework-core-1.0.0.jar:1.0.0]
... 4 common frames omitted
2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown Cluster Coordinator,5,main]: java.lang.NullPointerException
2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator] org.apache.nifi.NiFi
java.lang.NullPointerException: null
at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011) ~[na:1.8.0_51]
at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) ~[na:1.8.0_51]
at org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570) ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
at org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119) ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Failed to close Leader Selector for Cluster Coordinator
java.lang.IllegalStateException: Already closed or has not been started
at com.google.common.base.Preconditions.checkState(Preconditions.java:173) ~[guava-18.0.jar:na]
at org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270) ~[curator-recipes-2.11.0.jar:na]
at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303) [nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339) [nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753) [nifi-jetty-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.<init>(NiFi.java:152) [nifi-runtime-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.main(NiFi.java:243) [nifi-runtime-1.0.0.jar:1.0.0]
2016-10-19 13:14:45,062 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Socket closed
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method) ~[na:1.8.0_51]
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) ~[na:1.8.0_51]
at java.net.ServerSocket.implAccept(ServerSocket.java:545) ~[na:1.8.0_51]
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) ~[na:1.8.0_51]
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112) ~[nifi-socket-utils-1.0.0.jar:1.0.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.lang.Exception: Unable to load flow due to: java.io.IOException: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755) ~[nifi-jetty-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.<init>(NiFi.java:152) [nifi-runtime-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.main(NiFi.java:243) [nifi-runtime-1.0.0.jar:1.0.0]
Caused by: java.io.IOException: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) ~[nifi-jetty-1.0.0.jar:1.0.0]
... 2 common frames omitted
Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493) ~[nifi-framework-core-1.0.0.jar:1.0.0]
... 3 common frames omitted
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671) ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857) ~[nifi-framework-core-1.0.0.jar:1.0.0]
... 4 common frames omitted
[root@ncm-cm1 logs]#
I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is part of the problem too.
The docs weren’t very clear on whether (when upgrading and using the legacy support of the authorized-user.xml path required the nodes to be also added to the authorizers.xml.
I did add them in the end as various attempts to get the cluster up and running without them failed (as each server didn’t seem to have rights to do anything.
I have a lot of RPG in my work flows as I am ingesting many syslog data sources and this was the recommended pattern to distribute the data (listensyslog…run on primary, output to port (RPG), pick up in rest of data flow),
Any suggestions on where to start trying to get this working?
I’ve tried creating a new RPG on one on the datanodes and connecting the syslog to that which sort of worked but then I have a bunch of other errors when trying to enable the ports to do with not being able to connect to (what was) the NCM.
Thanks
Conrad
SecureData, combating cyber threats
______________________________________________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Hi,
Ok, so I have now connected my (old NCM) server to the cluster, but now I am getting reports in the other server logs that can’t connect due to SSL handshake exception…
2016-10-19 14:57:02,363 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@590db21a Unable to refresh Remote Group's peers due to Remote host closed connection during handshake
2016-10-19 14:57:02,379 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@4db0b1a7 Unable to refresh Remote Group's peers due to Remote host closed connection during handshake
2016-10-19 14:57:02,379 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@52ab2525 Unable to refresh Remote Group's peers due to Remote host closed connection during handshake
etc.etc.etc….
Prior to upgrade all servers ‘talked’ to each other no problem. This would suggest a problem with certs, but can’t see why this would be introduced as an issue with the upgrade.
For what its worth, I have followed the wiki article from way back about separating off new installs from a common root of directories (for repositories etc.).
Any other suggestions??
Thanks
Conrad
From: Bryan Bende <bb...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 19 October 2016 at 14:16
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Regarding the error about the uninheritable flow caused by the "proposed authorizations do not match current authorizations"... that basically means that the node trying to connect has a different set of authorizations (users/groups/policies) from what the other nodes in the cluster have.
This most likely means that something in the authorizers.xml on the new node is different from the authorizers.xml on the other nodes, and thus generated different users/groups/policies on that node during start up.
The process to add a new node to an existing cluster would be the following...
- From the UI, add a user for the DN of the new node
- Go to the new node and configure authorizers.xml so that it does not have an initial admin, legacy file, or node identities, by having none of this it will inherit everything from the cluster
- Start the new node
Since you already attempted to start this node, you will want to stop it and delete users.xml and authorizations.xml before attempting the above process.
Sorry if the documentation is not clear on this.
On Wed, Oct 19, 2016 at 9:12 AM, Joe Witt <jo...@gmail.com>> wrote:
Conrad,
For s2s now you can just point at any of the nodes in the cluster.
Have you tried changing the URL or removing and adding new RPG
entries?
Thanks
Joe
On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
<co...@secdata.com>> wrote:
> Hi,
>
> I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
>
> 6 nodes with a NCM.
>
> With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> my Remote Process Groups work as they previously did because they were
> configured to connect to the NCM (as the RPG url) which now doesn’t exist.
>
> I have tried converting my NCM to a node but whilst I can get it running
> (sort of) when I try and connect to the cluster I get something like this in
> my logs…
>
>
>
> 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> Failed to load flow from cluster due to:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> [nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> Authorizer is not inheritable by the flow controller because of Authorizer
> differences: Proposed Authorizations do not match current Authorizations
>
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 4 common frames omitted
>
> 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect node to cluster because local flow is different than cluster
> flow.
>
> 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> Cluster Coordinator,5,main]: java.lang.NullPointerException
>
> 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi
>
> java.lang.NullPointerException: null
>
> at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> ~[na:1.8.0_51]
>
> at
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> ~[na:1.8.0_51]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
>
> 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> Failed to close Leader Selector for Cluster Coordinator
>
> java.lang.IllegalStateException: Already closed or has not been started
>
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> ~[guava-18.0.jar:na]
>
> at
> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> ~[curator-recipes-2.11.0.jar:na]
>
> at
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> [nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> Host due to java.net.SocketException: Socket closed
>
> java.net.SocketException: Socket closed
>
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> ~[na:1.8.0_51]
>
> at
> java.net<http://java.net>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> ~[na:1.8.0_51]
>
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> ~[na:1.8.0_51]
>
> at
> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> ~[na:1.8.0_51]
>
> at
> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
>
> 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
>
> java.lang.Exception: Unable to load flow due to: java.io.IOException:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> ~[nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> Caused by: java.io.IOException:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> ~[nifi-jetty-1.0.0.jar:1.0.0]
>
> ... 2 common frames omitted
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> connect node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 3 common frames omitted
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> Authorizer is not inheritable by the flow controller because of Authorizer
> differences: Proposed Authorizations do not match current Authorizations
>
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 4 common frames omitted
>
> [root@ncm-cm1 logs]#
>
>
>
> I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> part of the problem too.
>
> The docs weren’t very clear on whether (when upgrading and using the legacy
> support of the authorized-user.xml path required the nodes to be also added
> to the authorizers.xml.
>
> I did add them in the end as various attempts to get the cluster up and
> running without them failed (as each server didn’t seem to have rights to do
> anything.
>
>
>
> I have a lot of RPG in my work flows as I am ingesting many syslog data
> sources and this was the recommended pattern to distribute the data
> (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> flow),
>
>
>
> Any suggestions on where to start trying to get this working?
>
> I’ve tried creating a new RPG on one on the datanodes and connecting the
> syslog to that which sort of worked but then I have a bunch of other errors
> when trying to enable the ports to do with not being able to connect to
> (what was) the NCM.
>
>
>
> Thanks
>
> Conrad
>
>
>
> SecureData, combating cyber threats
>
> ________________________________
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email are
> those of the individual and not necessarily of SecureData Europe Ltd. Any
> prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT
***This email originated outside SecureData***
Click here<https://www.mailcontrol.com/sr/0ulpuIS9abTGX2PQPOmvUu5zZAYN1MosvyIBd3ccm1uXNGYSg1FTwTmtRvHb6XdL25LWvxYjTGeVGm43FW9a3A==> to report this email as spam.
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Bryan Bende <bb...@gmail.com>.
Regarding the error about the uninheritable flow caused by the "proposed
authorizations do not match current authorizations"... that basically means
that the node trying to connect has a different set of authorizations
(users/groups/policies) from what the other nodes in the cluster have.
This most likely means that something in the authorizers.xml on the new
node is different from the authorizers.xml on the other nodes, and thus
generated different users/groups/policies on that node during start up.
The process to add a new node to an existing cluster would be the
following...
- From the UI, add a user for the DN of the new node
- Go to the new node and configure authorizers.xml so that it does not have
an initial admin, legacy file, or node identities, by having none of this
it will inherit everything from the cluster
- Start the new node
Since you already attempted to start this node, you will want to stop it
and delete users.xml and authorizations.xml before attempting the above
process.
Sorry if the documentation is not clear on this.
On Wed, Oct 19, 2016 at 9:12 AM, Joe Witt <jo...@gmail.com> wrote:
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to
> 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where
> none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t
> exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like
> this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.
> StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > Authorizer is not inheritable by the flow controller because of
> Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(
> StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.
> NodeClusterCoordinator
> > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException:
> Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(
> ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.
> updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.
> shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(
> StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.
> CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.
> close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.
> stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(
> FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(
> StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with
> Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:
> 404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(
> SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.
> JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Failed to
> > connect node to cluster because local flow is different than cluster
> flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > Authorizer is not inheritable by the flow controller because of
> Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(
> StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception
> is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the
> legacy
> > support of the authorized-user.xml path required the nodes to be also
> added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights
> to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of
> data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the datanodes and connecting the
> > syslog to that which sort of worked but then I have a bunch of other
> errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may
> be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email
> are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
No problem – the efforts I have gone through pale into insignificance compared to the sterling work being done to produce such a fine product as NiFi I am sure!
;-)
On 24/10/2016, 13:49, "Joe Witt" <jo...@gmail.com> wrote:
Thanks for following up Conrad and sorry for the challenges you went
through getting there. We have to make this important feature easier
to setup.
Joe
On Mon, Oct 24, 2016 at 8:46 AM, Conrad Crampton
<co...@secdata.com> wrote:
> Hi,
>
> Yes, I think this is the issue.
>
> I have changed the nifi.remote.input.socket.host to be
> nifi.remote.input.host.
>
> I think I have over-thought the comment at the top of the admin docs for
> Site to Site
>
> “Properties named with nifi.remote.input.socket.* are RAW transport protocol
> specific. Similarly, nifi.remote.input.http.* are HTTP transport protocol
> specific properties.
>
> ” and made the leap of ‘logic’ to include
>
> a) Keep the ….input.socket.host from previous version
>
> b) Include a …input.http.host
>
> c) Include a nifi.remote.input.http.port
>
> I don’t suppose b & c matter too much as they are most likely ignored, but
> not having the correct setting for a) probably doesn’t help.
>
> With a vanilla install I still think not having http enables causes the NPE
> when using RAW.
>
> If someone can confirm if there is an outstanding JIRA for this (or fixed on
> master branch) then I won’t record another one. If I do, probably will be a
> bit garbled given the probably root cause as above.
>
>
>
> Thanks again for everyone’s input on this. I have now reverted back to my
> original 0.6.1 version then successfully upgraded to 1.0.0.
>
>
>
> Conrad
>
>
>
> From: Bryan Bende <bb...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Friday, 21 October 2016 at 16:09
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Conrad,
>
>
>
> I'm glad you were able to get something working. I am definitely confused by
> the properties that were required though.
>
>
>
> My understanding is that for RAW site-to-site it uses:
>
>
>
> nifi.remote.input.host=
>
> nifi.remote.input.socket.port=
>
>
>
> And for http based site-to-site it uses the web http or https port that you
> have configured (same as the UI).
>
>
>
> The properties you added seem to be slightly different. One thing I noticed
> was that nifi.remote.input.socket.host changed to nifi.remote.input.host in
> 1.0.0, so maybe that was a problem.
>
>
>
> The latest base nifi.properties is here:
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties
>
>
>
> On Friday, October 21, 2016, Joe Witt <jo...@gmail.com> wrote:
>
> It's ok. If you see it again would love to get the stack trace. NPEs
> automatically should turn into JIRAs as they should definitely never
> happen. I'm not suggesting tracking this down will help your s2s/rpg
> struggles but there is clearly something we can tighten up there.
>
> On Fri, Oct 21, 2016 at 10:43 AM, Conrad Crampton
> <co...@secdata.com> wrote:
>> I’ve just tried to replicate but without success. Using a RPG with RAW
>> transfer protocol.
>>
>> If you want I can try removing the additional lines in my nifi.properties
>> (the ones for http.port and host that I added which appeared to get this
>> working?
>>
>> Regards
>> Conrad
>>
>>
>> On 21/10/2016, 15:24, "Joe Witt" <jo...@gmail.com> wrote:
>>
>> Whoa there.... can you turn on debug logging in conf/logback.xml by
>> adding
>>
>> <logger name="org.apache.nifi.remote.SocketRemoteSiteListener "
>> level="DEBUG"/>
>>
>> Can you submit a JIRA for the log output with the stacktrace for those
>> NPEs please.
>>
>> Thanks
>> Joe
>>
>> On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
>> <co...@secdata.com> wrote:
>> > Hi,
>> >
>> > I realise this may be getting boring for most but…
>> >
>> > I didn’t find any resolution to the upgrade so I have cleanly
>> installed
>> > v1.0.0 and oddly experienced the same issue with RPGs in that
>> although the
>> > RPGs didn’t show any errors (in so much as they had no warning on
>> them and
>> > reported that S2S was secure) the errors reported were about not
>> being able
>> > to determine other nodes in cluster.
>> >
>> > This is a section of log that showed an error that I don’t think I
>> saw
>> > before but only including here in case one of you fine folks need it
>> for any
>> > reason…
>> >
>> >
>> >
>> > ERROR [Site-to-Site Worker Thread-194]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode3-cm1.mis.local:57289]
>> >
>> (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c8-bffc59d650a7])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> > 2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode1-cm1.mis.local:35831]
>> >
>> (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c9-d294bb4037f5])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> > 2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode5-cm1.mis.local:59093]
>> >
>> (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-8201-16b905e5beb6])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> > 2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode6-cm1.mis.local:37275]
>> >
>> (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a149-4892d0193819])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> > 2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode4-cm1.mis.local:57745]
>> >
>> (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-8047-b5ba4d9b67ec])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> >
>> >
>> > in my naivety this suggests a problem with the socket (RAW)
>> connection
>> > protocol. The relevant section for S2S connection in my
>> nifi.properties is
>> >
>> > nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the number
>> > different for each node obviously
>> >
>> > nifi.remote.input.socket.port=10443
>> >
>> > nifi.remote.input.secure=true
>> >
>> > nifi.remote.input.http.enabled=false
>> >
>> > nifi.remote.input.http.transaction.ttl=30 sec
>> >
>> >
>> >
>> > the errors suggest that the port specified here aren’t being used,
>> but some
>> > random ports are being used instead. Of course this is complete
>> supposition
>> > and probably a red herring.
>> >
>> >
>> >
>> > Anyway, I updated my nifi.properties to
>> >
>> > nifi.remote.input.socket.host=nifinode6-cm1.mis.local
>> >
>> > nifi.remote.input.http.host=nifinode6-cm1.mis.local
>> >
>> > nifi.remote.input.socket.port=10443
>> >
>> > nifi.remote.input.http.port=11443
>> >
>> > nifi.remote.input.secure=true
>> >
>> > nifi.remote.input.http.enabled=true
>> >
>> > nifi.remote.input.http.transaction.ttl=30 sec
>> >
>> >
>> >
>> > and used HTTP for my RPG and is now working ok.
>> >
>> >
>> >
>> > The test harness for this is
>> >
>> >
>> >
>> > GenerateFlowFile->RGP(test input port)
>> >
>> > InputPort(test input)->LogAttribute
>> >
>> >
>> >
>> > Regards,
>> >
>> > Conrad
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > From: Conrad Crampton <co...@SecData.com>
>> > Date: Friday, 21 October 2016 at 08:18
>> >
>> >
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> > Hi,
>> >
>> > Yes, the access policy for the input port at the root level of my
>> workspace
>> > has S2S access policy (receive data via site to site) for all server
>> nodes
>> > (I have all my nodes in one group).
>> >
>> > For the next level down in my workspace (I have other process groups
>> from my
>> > main ‘root’ page in the UI space to organise the flows), I have
>> input ports
>> > on the next level down of flows which when I check the access
>> policies on
>> > that for S2S, the receive and send data via site-to-site options are
>> greyed
>> > out and if I try to override the policy, they still are. I don’t
>> know if
>> > this is important, but from reading the post below, the issue that
>> the
>> > access policies looks to address is different from my issue. The RPG
>> has a
>> > locked padlock and says site to site is secure. I don’t have any
>> warning
>> > triangles on the RPG itself, but I have the aforementioned warnings
>> in my
>> > logs.
>> >
>> >
>> >
>> > I’m going to abandon this now as I can’t get it to work.
>> >
>> >
>> >
>> > I’m going to try a fresh install and go with that – and have to
>> recreate all
>> > my flows where necessary. I’m moving to a new model of data
>> ingestion anyway
>> > so isn’t as catastrophic as it might be.
>> >
>> >
>> >
>> > Thanks for the help.
>> >
>> > Conrad
>> >
>> >
>> >
>> > From: Andy LoPresto <al...@apache.org>
>> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Date: Thursday, 20 October 2016 at 17:31
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> > * PGP Signed by an unknown key
>> >
>> > Conrad,
>> >
>> >
>> >
>> > For the site-to-site did you follow the instructions here [1]? Each
>> node
>> > needs to be added as a user in order to make the connections.
>> >
>> >
>> >
>> > [1]
>> >
>> http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>> >
>> >
>> >
>> > Andy LoPresto
>> >
>> > alopresto@apache.org
>> >
>> > alopresto.apache@gmail.com
>> >
>> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >
>> >
>> >
>> > On Oct 20, 2016, at 7:36 AM, Conrad Crampton
>> <co...@SecData.com>
>> > wrote:
>> >
>> >
>> >
>> > Ok,
>> >
>> > So I tried everything suggested so far to no avail unfortunately.
>> >
>> >
>> >
>> > So what I have done is to create all new certs etc. using the
>> tookit.
>> > Updated my existing authoriszed-users.xml to have to match the full
>> cert
>> > distinguished names CN=server.name, OU=NIFI etc.
>> >
>> >
>> >
>> > Recreated all my remote process groups to not reference the original
>> NCM as
>> > that still wouldn’t work – after a complete new install (upgrade).
>> >
>> >
>> >
>> > So now what I have is a six node cluster using original data/worker
>> nodes
>> > and they are part of the cluster – all appears to be working ie. I
>> can log
>> > into the UI (nice by the way ;-) on each server. There are no SSL
>> handshake
>> > errors etc. BUT the RPG (newly created) still don’t appear to be
>> working. I
>> > am getting
>> >
>> >
>> >
>> > 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>> >
>> >
>> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>> >
>> >
>> nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>> >
>> >
>> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>> >
>> >
>> nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>> >
>> >
>> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>> >
>> >
>> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>> >
>> >
>> nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> >
>> >
>> > I can telnet from server to server on both https (UI) port and S2S
>> port.
>> >
>> > I am really at a loss as to what to do now.
>> >
>> >
>> >
>> > Data is queuing up in my input processors with nowhere to go.
>> >
>> > Do I have to do something radical here to get this working like
>> stopping
>> > everything, clearing out all the queues then starting up again??? I
>> really
>> > don’t want to do this obviously but I am getting nowhere on this –
>> two days
>> > of frustration with nothing to show for it.
>> >
>> >
>> >
>> > Any more suggestions please??
>> >
>> > Thanks for your patience.
>> >
>> > Conrad
>> >
>> >
>> >
>> >
>> >
>> > From: Andy LoPresto <al...@apache.org>
>> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Date: Wednesday, 19 October 2016 at 18:24
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> >> Old Signed by an unknown key
>> >
>> > Hi Conrad,
>> >
>> >
>> >
>> > Bryan is correct that changing the certificates (and the
>> encapsulating
>> > keystores and truststores) will not affect any data held in the
>> nodes.
>> >
>> >
>> >
>> > Regenerating everything using the TLS toolkit should hopefully not
>> be too
>> > challenging, but I am also curious as to why you are getting these
>> handshake
>> > exceptions now. As Bryan pointed out, adding the following line to
>> > bootstrap.conf will provide substantial additional log output which
>> should
>> > help trace the issue.
>> >
>> >
>> >
>> > java.arg.15=-Djavax.net.debug=ssl,handshake
>> >
>> >
>> >
>> > You can also imitate the node connecting to the (previous) NCM via
>> this
>> > command:
>> >
>> >
>> >
>> > $ openssl s_client -connect <host:port> -debug -state -cert
>> > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>> > <path_to_your_CA_cert.pem>
>> >
>> >
>> >
>> > Where:
>> >
>> >
>> >
>> > <host:port> = the hostname and port of the “NCM”
>> > <path_to_your_cert.pem> = the public key used to identify the “node”
>> (can be
>> > exported from the node keystore [1])
>> > <path_to_your_key.pem> = the private key used to identify the “node”
>> (can be
>> > exported from the node keystore via 2 step process)
>> > <path_to_your_CA_cert.pem> = the public key used to sign the “NCM”
>> > certificate (could be a 3rd party like Verisign or DigiCert, or an
>> internal
>> > organization CA if you have one)
>> >
>> >
>> >
>> > If you’ve already regenerated everything and it works, that’s fine.
>> But if
>> > you have the time to try and investigate the old certs, we are
>> interested
>> > and prepared to help. Thanks.
>> >
>> >
>> >
>> > [1] https://security.stackexchange.com/a/66865/16485
>> >
>> >
>> >
>> > Andy LoPresto
>> >
>> > alopresto@apache.org
>> >
>> > alopresto.apache@gmail.com
>> >
>> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >
>> >
>> >
>> > On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com> wrote:
>> >
>> >
>> >
>> > That is definitely weird that it is only an issue on the node that
>> used to
>> > be the NCM.
>> >
>> > Might be worth double checking the keystore and truststore of that
>> one node
>> > and make sure it has what you would expect, and also double check
>> > nifi.properties compared to the others to see if anything seems
>> different.
>> >
>> >
>> >
>> > Changing all of the keystores, truststores, etc should be fine from
>> a data
>> > perspective...
>> >
>> >
>> >
>> > If you decide to go that route it would probably be easiest to start
>> back
>> > over from a security perspective, meaning:
>> >
>> > - Stop all the nodes and delete the users.xml and authorizations.xml
>> from
>> > all nodes
>> >
>> > - Configure authorizers.xml with the appropriate initial admin (or
>> legacy
>> > file) and node identities based on the new certs
>> >
>> > - Ensure authorizers.xml is the same on all nodes
>> >
>> > - Then restart everything
>> >
>> >
>> >
>> > Alternatively, you might be able to manually add users for all of
>> the new
>> > certs before shutting everything down and give them the appropriate
>> > policies, then restart everything, but requires more manual work to
>> get
>> > everything lined up.
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton
>> > <co...@secdata.com> wrote:
>> >
>> > Hi,
>> >
>> > As a plan for tomorrow – I have generated new keystores,
>> truststores, client
>> > certts etc. for all nodes in my cluster using the
>> >
>> >
>> >
>> > From: Bryan Bende <bb...@gmail.com>
>> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Date: Wednesday, 19 October 2016 at 15:33
>> >
>> >
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> >
>> >
>> > Trying to think of things to check here...
>> >
>> >
>> >
>> > Does every node have nifi.remote.input.secure=true in
>> nifi.properties and
>> > the URL in the RPG is an https URL?
>> >
>> >
>> >
>> > On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton
>> > <co...@secdata.com> wrote:
>> >
>> > One other thing…
>> >
>> > The RPGs have an unlocked padlock on them saying S2S is not secure.
>> >
>> > Conrad
>> >
>> >
>> >
>> > From: Bryan Bende <bb...@gmail.com>
>> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Date: Wednesday, 19 October 2016 at 15:20
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> > Ok that does seem like a TLS/SSL issue...
>> >
>> >
>> >
>> > Is this a single cluster doing site-to-site to itself?
>> >
>> >
>> >
>> > On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>
>> wrote:
>> >
>> > thanks conrad - did get it. Bryan is being more helpful that I so I
>> > went silent :-)
>> >
>> > On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>> >
>> > <co...@secdata.com> wrote:
>> >> Hi Joe,
>> >> Yep,
>> >> Tried removing the RPG that referenced the NCM and adding new
>> one with
>> >> one of the nifis as url.
>> >> That sort of worked, but kept getting errors about the NCM not
>> being
>> >> available for the ports and therefore couldn’t actually enable the
>> port I
>> >> needed to for that RPG.
>> >> Thanks
>> >> Conrad
>> >>
>> >> (sending again as don’t know if the stupid header ‘spoofed’ is
>> stopping
>> >> getting though – apologies if already sent)
>> >>
>> >> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
>> >>
>> >> Conrad,
>> >>
>> >> For s2s now you can just point at any of the nodes in the
>> cluster.
>> >> Have you tried changing the URL or removing and adding new
>> RPG
>> >> entries?
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
>> >> <co...@secdata.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I have finally taken the plunge to upgrade my cluster
>> from 0.6.1
>> >> to 1.0.0.
>> >> >
>> >> > 6 nodes with a NCM.
>> >> >
>> >> > With the removal of NCM in 1.0.0 I believe I now have an
>> issue
>> >> where none of
>> >> > my Remote Process Groups work as they previously did
>> because
>> >> they were
>> >> > configured to connect to the NCM (as the RPG url) which
>> now
>> >> doesn’t exist.
>> >> >
>> >> > I have tried converting my NCM to a node but whilst I can
>> get it
>> >> running
>> >> > (sort of) when I try and connect to the cluster I get
>> something
>> >> like this in
>> >> > my logs…
>> >> >
>> >> >
>> >> >
>> >> > 2016-10-19 13:14:44,109 ERROR [main]
>> >> o.a.nifi.controller.StandardFlowService
>> >> > Failed to load flow from cluster due to:
>> >> > org.apache.nifi.controller.UninheritableFlowException:
>> Failed to
>> >> connect
>> >> > node to cluster because local flow is different than
>> cluster
>> >> flow.
>> >> >
>> >> > org.apache.nifi.controller.UninheritableFlowException:
>> Failed to
>> >> connect
>> >> > node to cluster because local flow is different than
>> cluster
>> >> flow.
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> >> > [nifi-jetty-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > Caused by:
>> >> org.apache.nifi.controller.UninheritableFlowException: Proposed
>> >> > Authorizer is not inheritable by the flow controller
>> because of
>> >> Authorizer
>> >> > differences: Proposed Authorizations do not match current
>> >> Authorizations
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > ... 4 common frames omitted
>> >> >
>> >> > 2016-10-19 13:14:44,414 ERROR [main]
>> >> o.a.n.c.c.node.NodeClusterCoordinator
>> >> > Event Reported for ncm-cm1.local:9090 -- Node
>> disconnected from
>> >> > cluster due to
>> >> org.apache.nifi.controller.UninheritableFlowException: Failed
>> >> > to connect node to cluster because local flow is
>> different than
>> >> cluster
>> >> > flow.
>> >> >
>> >> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster
>> Coordinator]
>> >> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
>> >> Thread[Shutdown
>> >> > Cluster Coordinator,5,main]:
>> java.lang.NullPointerException
>> >> >
>> >> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster
>> Coordinator]
>> >> > org.apache.nifi.NiFi
>> >> >
>> >> > java.lang.NullPointerException: null
>> >> >
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
>> >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
>> >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at java.lang.Thread.run(Thread.java:745)
>> ~[na:1.8.0_51]
>> >> >
>> >> > 2016-10-19 13:14:44,448 WARN [main]
>> >> o.a.n.c.l.e.CuratorLeaderElectionManager
>> >> > Failed to close Leader Selector for Cluster Coordinator
>> >> >
>> >> > java.lang.IllegalStateException: Already closed or has
>> not been
>> >> started
>> >> >
>> >> > at
>> >> >
>> >>
>> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
>> >> > ~[guava-18.0.jar:na]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
>> >> > ~[curator-recipes-2.11.0.jar:na]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
>> >> > [nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
>> >> > [nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
>> >> > [nifi-jetty-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
>> >> > org.apache.nifi.io.socket.SocketListener Failed to
>> communicate
>> >> with Unknown
>> >> > Host due to java.net.SocketException: Socket closed
>> >> >
>> >> > java.net.SocketException: Socket closed
>> >> >
>> >> > at java.net.PlainSocketImpl.socketAccept(Native
>> Method)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
>> >> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>> >> >
>> >> > at java.lang.Thread.run(Thread.java:745)
>> [na:1.8.0_51]
>> >> >
>> >> > 2016-10-19 13:14:45,064 WARN [main]
>> >> org.apache.nifi.web.server.JettyServer
>> >> > Failed to start web server... shutting down.
>> >> >
>> >> > java.lang.Exception: Unable to load flow due to:
>> >> java.io.IOException:
>> >> > org.apache.nifi.controller.UninheritableFlowException:
>> Failed to
>> >> connect
>> >> > node to cluster because local flow is different than
>> cluster
>> >> flow.
>> >> >
>> >> > at
>> >> >
>> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
>> >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > Caused by: java.io.IOException:
>> >> > org.apache.nifi.controller.UninheritableFlowException:
>> Failed to
>> >> connect
>> >> > node to cluster because local flow is different than
>> cluster
>> >> flow.
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> >> >
>> >> > ... 2 common frames omitted
>> >> >
>> >> > Caused by:
>> >> org.apache.nifi.controller.UninheritableFlowException: Failed to
>> >> > connect node to cluster because local flow is different
>> than
>> >> cluster flow.
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > ... 3 common frames omitted
>> >> >
>> >> > Caused by:
>> >> org.apache.nifi.controller.UninheritableFlowException: Proposed
>> >> > Authorizer is not inheritable by the flow controller
>> because of
>> >> Authorizer
>> >> > differences: Proposed Authorizations do not match current
>> >> Authorizations
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > ... 4 common frames omitted
>> >> >
>> >> > [root@ncm-cm1 logs]#
>> >> >
>> >> >
>> >> >
>> >> > I don’t know if the ‘Proposed Authorizer is not
>> inheritable…’
>> >> exception is
>> >> > part of the problem too.
>> >> >
>> >> > The docs weren’t very clear on whether (when upgrading
>> and using
>> >> the legacy
>> >> > support of the authorized-user.xml path required the
>> nodes to be
>> >> also added
>> >> > to the authorizers.xml.
>> >> >
>> >> > I did add them in the end as various attempts to get the
>> cluster
>> >> up and
>> >> > running without them failed (as each server didn’t seem
>> to have
>> >> rights to do
>> >> > anything.
>> >> >
>> >> >
>> >> >
>> >> > I have a lot of RPG in my work flows as I am ingesting
>> many
>> >> syslog data
>> >> > sources and this was the recommended pattern to
>> distribute the
>> >> data
>> >> > (listensyslog…run on primary, output to port (RPG), pick
>> up in
>> >> rest of data
>> >> > flow),
>> >> >
>> >> >
>> >> >
>> >> > Any suggestions on where to start trying to get this
>> working?
>> >> >
>> >> > I’ve tried creating a new RPG on one on the nifis and
>> connecting
>> >> the
>> >> > syslog to that which sort of worked but then I have a
>> bunch of
>> >> other errors
>> >> > when trying to enable the ports to do with not being able
>> to
>> >> connect to
>> >> > (what was) the NCM.
>> >> >
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Conrad
>> >> >
>> >> >
>> >> >
>> >> > SecureData, combating cyber threats
>> >> >
>> >> > ________________________________
>> >> >
>> >> > The information contained in this message or any of its
>> >> attachments may be
>> >> > privileged and confidential and intended for the
>> exclusive use
>> >> of the
>> >> > intended recipient. If you are not the intended recipient
>> any
>> >> disclosure,
>> >> > reproduction, distribution or other dissemination or use
>> of this
>> >> > communications is strictly prohibited. The views
>> expressed in
>> >> this email are
>> >> > those of the individual and not necessarily of SecureData
>> Europe
>> >> Ltd. Any
>> >> > prices quoted are only valid if followed up by a formal
>> written
>> >> quote.
>> >> >
>> >> > SecureData Europe Limited. Registered in England & Wales
>> >> 04365896.
>> >> > Registered Address: SecureData House, Hermitage Court,
>> Hermitage
>> >> Lane,
>> >> > Maidstone, Kent, ME16 9NT
>> >>
>> >>
>> >> ***This email originated outside SecureData***
>> >>
>> >> Click
>> >>
>> https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==
>> >> to report this email as spam.
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > * Unknown Key
>> > * 0x2F7DEF69
>> >
>> >
>> >
>> >
>> >
>> > * Unknown Key
>> > * 0x2F7DEF69
>> >
>> >
>>
>>
>
>
>
> --
> Sent from Gmail Mobile
>
SecureData, combating cyber threats
______________________________________________________________________
The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote.
SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Joe Witt <jo...@gmail.com>.
Thanks for following up Conrad and sorry for the challenges you went
through getting there. We have to make this important feature easier
to setup.
Joe
On Mon, Oct 24, 2016 at 8:46 AM, Conrad Crampton
<co...@secdata.com> wrote:
> Hi,
>
> Yes, I think this is the issue.
>
> I have changed the nifi.remote.input.socket.host to be
> nifi.remote.input.host.
>
> I think I have over-thought the comment at the top of the admin docs for
> Site to Site
>
> “Properties named with nifi.remote.input.socket.* are RAW transport protocol
> specific. Similarly, nifi.remote.input.http.* are HTTP transport protocol
> specific properties.
>
> ” and made the leap of ‘logic’ to include
>
> a) Keep the ….input.socket.host from previous version
>
> b) Include a …input.http.host
>
> c) Include a nifi.remote.input.http.port
>
> I don’t suppose b & c matter too much as they are most likely ignored, but
> not having the correct setting for a) probably doesn’t help.
>
> With a vanilla install I still think not having http enables causes the NPE
> when using RAW.
>
> If someone can confirm if there is an outstanding JIRA for this (or fixed on
> master branch) then I won’t record another one. If I do, probably will be a
> bit garbled given the probably root cause as above.
>
>
>
> Thanks again for everyone’s input on this. I have now reverted back to my
> original 0.6.1 version then successfully upgraded to 1.0.0.
>
>
>
> Conrad
>
>
>
> From: Bryan Bende <bb...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Friday, 21 October 2016 at 16:09
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Conrad,
>
>
>
> I'm glad you were able to get something working. I am definitely confused by
> the properties that were required though.
>
>
>
> My understanding is that for RAW site-to-site it uses:
>
>
>
> nifi.remote.input.host=
>
> nifi.remote.input.socket.port=
>
>
>
> And for http based site-to-site it uses the web http or https port that you
> have configured (same as the UI).
>
>
>
> The properties you added seem to be slightly different. One thing I noticed
> was that nifi.remote.input.socket.host changed to nifi.remote.input.host in
> 1.0.0, so maybe that was a problem.
>
>
>
> The latest base nifi.properties is here:
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties
>
>
>
> On Friday, October 21, 2016, Joe Witt <jo...@gmail.com> wrote:
>
> It's ok. If you see it again would love to get the stack trace. NPEs
> automatically should turn into JIRAs as they should definitely never
> happen. I'm not suggesting tracking this down will help your s2s/rpg
> struggles but there is clearly something we can tighten up there.
>
> On Fri, Oct 21, 2016 at 10:43 AM, Conrad Crampton
> <co...@secdata.com> wrote:
>> I’ve just tried to replicate but without success. Using a RPG with RAW
>> transfer protocol.
>>
>> If you want I can try removing the additional lines in my nifi.properties
>> (the ones for http.port and host that I added which appeared to get this
>> working?
>>
>> Regards
>> Conrad
>>
>>
>> On 21/10/2016, 15:24, "Joe Witt" <jo...@gmail.com> wrote:
>>
>> Whoa there.... can you turn on debug logging in conf/logback.xml by
>> adding
>>
>> <logger name="org.apache.nifi.remote.SocketRemoteSiteListener "
>> level="DEBUG"/>
>>
>> Can you submit a JIRA for the log output with the stacktrace for those
>> NPEs please.
>>
>> Thanks
>> Joe
>>
>> On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
>> <co...@secdata.com> wrote:
>> > Hi,
>> >
>> > I realise this may be getting boring for most but…
>> >
>> > I didn’t find any resolution to the upgrade so I have cleanly
>> installed
>> > v1.0.0 and oddly experienced the same issue with RPGs in that
>> although the
>> > RPGs didn’t show any errors (in so much as they had no warning on
>> them and
>> > reported that S2S was secure) the errors reported were about not
>> being able
>> > to determine other nodes in cluster.
>> >
>> > This is a section of log that showed an error that I don’t think I
>> saw
>> > before but only including here in case one of you fine folks need it
>> for any
>> > reason…
>> >
>> >
>> >
>> > ERROR [Site-to-Site Worker Thread-194]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode3-cm1.mis.local:57289]
>> >
>> (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c8-bffc59d650a7])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> > 2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode1-cm1.mis.local:35831]
>> >
>> (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c9-d294bb4037f5])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> > 2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode5-cm1.mis.local:59093]
>> >
>> (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-8201-16b905e5beb6])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> > 2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode6-cm1.mis.local:37275]
>> >
>> (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a149-4892d0193819])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> > 2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198]
>> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with
>> remote
>> > instance Peer[url=nifi://nifinode4-cm1.mis.local:57745]
>> >
>> (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-8047-b5ba4d9b67ec])
>> > due to java.lang.NullPointerException; closing connection
>> >
>> >
>> >
>> > in my naivety this suggests a problem with the socket (RAW)
>> connection
>> > protocol. The relevant section for S2S connection in my
>> nifi.properties is
>> >
>> > nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the number
>> > different for each node obviously
>> >
>> > nifi.remote.input.socket.port=10443
>> >
>> > nifi.remote.input.secure=true
>> >
>> > nifi.remote.input.http.enabled=false
>> >
>> > nifi.remote.input.http.transaction.ttl=30 sec
>> >
>> >
>> >
>> > the errors suggest that the port specified here aren’t being used,
>> but some
>> > random ports are being used instead. Of course this is complete
>> supposition
>> > and probably a red herring.
>> >
>> >
>> >
>> > Anyway, I updated my nifi.properties to
>> >
>> > nifi.remote.input.socket.host=nifinode6-cm1.mis.local
>> >
>> > nifi.remote.input.http.host=nifinode6-cm1.mis.local
>> >
>> > nifi.remote.input.socket.port=10443
>> >
>> > nifi.remote.input.http.port=11443
>> >
>> > nifi.remote.input.secure=true
>> >
>> > nifi.remote.input.http.enabled=true
>> >
>> > nifi.remote.input.http.transaction.ttl=30 sec
>> >
>> >
>> >
>> > and used HTTP for my RPG and is now working ok.
>> >
>> >
>> >
>> > The test harness for this is
>> >
>> >
>> >
>> > GenerateFlowFile->RGP(test input port)
>> >
>> > InputPort(test input)->LogAttribute
>> >
>> >
>> >
>> > Regards,
>> >
>> > Conrad
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > From: Conrad Crampton <co...@SecData.com>
>> > Date: Friday, 21 October 2016 at 08:18
>> >
>> >
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> > Hi,
>> >
>> > Yes, the access policy for the input port at the root level of my
>> workspace
>> > has S2S access policy (receive data via site to site) for all server
>> nodes
>> > (I have all my nodes in one group).
>> >
>> > For the next level down in my workspace (I have other process groups
>> from my
>> > main ‘root’ page in the UI space to organise the flows), I have
>> input ports
>> > on the next level down of flows which when I check the access
>> policies on
>> > that for S2S, the receive and send data via site-to-site options are
>> greyed
>> > out and if I try to override the policy, they still are. I don’t
>> know if
>> > this is important, but from reading the post below, the issue that
>> the
>> > access policies looks to address is different from my issue. The RPG
>> has a
>> > locked padlock and says site to site is secure. I don’t have any
>> warning
>> > triangles on the RPG itself, but I have the aforementioned warnings
>> in my
>> > logs.
>> >
>> >
>> >
>> > I’m going to abandon this now as I can’t get it to work.
>> >
>> >
>> >
>> > I’m going to try a fresh install and go with that – and have to
>> recreate all
>> > my flows where necessary. I’m moving to a new model of data
>> ingestion anyway
>> > so isn’t as catastrophic as it might be.
>> >
>> >
>> >
>> > Thanks for the help.
>> >
>> > Conrad
>> >
>> >
>> >
>> > From: Andy LoPresto <al...@apache.org>
>> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Date: Thursday, 20 October 2016 at 17:31
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> > * PGP Signed by an unknown key
>> >
>> > Conrad,
>> >
>> >
>> >
>> > For the site-to-site did you follow the instructions here [1]? Each
>> node
>> > needs to be added as a user in order to make the connections.
>> >
>> >
>> >
>> > [1]
>> >
>> http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>> >
>> >
>> >
>> > Andy LoPresto
>> >
>> > alopresto@apache.org
>> >
>> > alopresto.apache@gmail.com
>> >
>> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >
>> >
>> >
>> > On Oct 20, 2016, at 7:36 AM, Conrad Crampton
>> <co...@SecData.com>
>> > wrote:
>> >
>> >
>> >
>> > Ok,
>> >
>> > So I tried everything suggested so far to no avail unfortunately.
>> >
>> >
>> >
>> > So what I have done is to create all new certs etc. using the
>> tookit.
>> > Updated my existing authoriszed-users.xml to have to match the full
>> cert
>> > distinguished names CN=server.name, OU=NIFI etc.
>> >
>> >
>> >
>> > Recreated all my remote process groups to not reference the original
>> NCM as
>> > that still wouldn’t work – after a complete new install (upgrade).
>> >
>> >
>> >
>> > So now what I have is a six node cluster using original data/worker
>> nodes
>> > and they are part of the cluster – all appears to be working ie. I
>> can log
>> > into the UI (nice by the way ;-) on each server. There are no SSL
>> handshake
>> > errors etc. BUT the RPG (newly created) still don’t appear to be
>> working. I
>> > am getting
>> >
>> >
>> >
>> > 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>> >
>> >
>> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>> >
>> >
>> nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>> >
>> >
>> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>> >
>> >
>> nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>> >
>> >
>> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>> >
>> >
>> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> > 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>> >
>> >
>> nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426
>> > Unable to refresh Remote Group's peers due to Unable to communicate
>> with
>> > remote NiFi cluster in order to determine which nodes exist in the
>> remote
>> > cluster
>> >
>> >
>> >
>> > I can telnet from server to server on both https (UI) port and S2S
>> port.
>> >
>> > I am really at a loss as to what to do now.
>> >
>> >
>> >
>> > Data is queuing up in my input processors with nowhere to go.
>> >
>> > Do I have to do something radical here to get this working like
>> stopping
>> > everything, clearing out all the queues then starting up again??? I
>> really
>> > don’t want to do this obviously but I am getting nowhere on this –
>> two days
>> > of frustration with nothing to show for it.
>> >
>> >
>> >
>> > Any more suggestions please??
>> >
>> > Thanks for your patience.
>> >
>> > Conrad
>> >
>> >
>> >
>> >
>> >
>> > From: Andy LoPresto <al...@apache.org>
>> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Date: Wednesday, 19 October 2016 at 18:24
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> >> Old Signed by an unknown key
>> >
>> > Hi Conrad,
>> >
>> >
>> >
>> > Bryan is correct that changing the certificates (and the
>> encapsulating
>> > keystores and truststores) will not affect any data held in the
>> nodes.
>> >
>> >
>> >
>> > Regenerating everything using the TLS toolkit should hopefully not
>> be too
>> > challenging, but I am also curious as to why you are getting these
>> handshake
>> > exceptions now. As Bryan pointed out, adding the following line to
>> > bootstrap.conf will provide substantial additional log output which
>> should
>> > help trace the issue.
>> >
>> >
>> >
>> > java.arg.15=-Djavax.net.debug=ssl,handshake
>> >
>> >
>> >
>> > You can also imitate the node connecting to the (previous) NCM via
>> this
>> > command:
>> >
>> >
>> >
>> > $ openssl s_client -connect <host:port> -debug -state -cert
>> > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>> > <path_to_your_CA_cert.pem>
>> >
>> >
>> >
>> > Where:
>> >
>> >
>> >
>> > <host:port> = the hostname and port of the “NCM”
>> > <path_to_your_cert.pem> = the public key used to identify the “node”
>> (can be
>> > exported from the node keystore [1])
>> > <path_to_your_key.pem> = the private key used to identify the “node”
>> (can be
>> > exported from the node keystore via 2 step process)
>> > <path_to_your_CA_cert.pem> = the public key used to sign the “NCM”
>> > certificate (could be a 3rd party like Verisign or DigiCert, or an
>> internal
>> > organization CA if you have one)
>> >
>> >
>> >
>> > If you’ve already regenerated everything and it works, that’s fine.
>> But if
>> > you have the time to try and investigate the old certs, we are
>> interested
>> > and prepared to help. Thanks.
>> >
>> >
>> >
>> > [1] https://security.stackexchange.com/a/66865/16485
>> >
>> >
>> >
>> > Andy LoPresto
>> >
>> > alopresto@apache.org
>> >
>> > alopresto.apache@gmail.com
>> >
>> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >
>> >
>> >
>> > On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com> wrote:
>> >
>> >
>> >
>> > That is definitely weird that it is only an issue on the node that
>> used to
>> > be the NCM.
>> >
>> > Might be worth double checking the keystore and truststore of that
>> one node
>> > and make sure it has what you would expect, and also double check
>> > nifi.properties compared to the others to see if anything seems
>> different.
>> >
>> >
>> >
>> > Changing all of the keystores, truststores, etc should be fine from
>> a data
>> > perspective...
>> >
>> >
>> >
>> > If you decide to go that route it would probably be easiest to start
>> back
>> > over from a security perspective, meaning:
>> >
>> > - Stop all the nodes and delete the users.xml and authorizations.xml
>> from
>> > all nodes
>> >
>> > - Configure authorizers.xml with the appropriate initial admin (or
>> legacy
>> > file) and node identities based on the new certs
>> >
>> > - Ensure authorizers.xml is the same on all nodes
>> >
>> > - Then restart everything
>> >
>> >
>> >
>> > Alternatively, you might be able to manually add users for all of
>> the new
>> > certs before shutting everything down and give them the appropriate
>> > policies, then restart everything, but requires more manual work to
>> get
>> > everything lined up.
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton
>> > <co...@secdata.com> wrote:
>> >
>> > Hi,
>> >
>> > As a plan for tomorrow – I have generated new keystores,
>> truststores, client
>> > certts etc. for all nodes in my cluster using the
>> >
>> >
>> >
>> > From: Bryan Bende <bb...@gmail.com>
>> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Date: Wednesday, 19 October 2016 at 15:33
>> >
>> >
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> >
>> >
>> > Trying to think of things to check here...
>> >
>> >
>> >
>> > Does every node have nifi.remote.input.secure=true in
>> nifi.properties and
>> > the URL in the RPG is an https URL?
>> >
>> >
>> >
>> > On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton
>> > <co...@secdata.com> wrote:
>> >
>> > One other thing…
>> >
>> > The RPGs have an unlocked padlock on them saying S2S is not secure.
>> >
>> > Conrad
>> >
>> >
>> >
>> > From: Bryan Bende <bb...@gmail.com>
>> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Date: Wednesday, 19 October 2016 at 15:20
>> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
>> Groups
>> >
>> >
>> >
>> > Ok that does seem like a TLS/SSL issue...
>> >
>> >
>> >
>> > Is this a single cluster doing site-to-site to itself?
>> >
>> >
>> >
>> > On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>
>> wrote:
>> >
>> > thanks conrad - did get it. Bryan is being more helpful that I so I
>> > went silent :-)
>> >
>> > On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>> >
>> > <co...@secdata.com> wrote:
>> >> Hi Joe,
>> >> Yep,
>> >> Tried removing the RPG that referenced the NCM and adding new
>> one with
>> >> one of the nifis as url.
>> >> That sort of worked, but kept getting errors about the NCM not
>> being
>> >> available for the ports and therefore couldn’t actually enable the
>> port I
>> >> needed to for that RPG.
>> >> Thanks
>> >> Conrad
>> >>
>> >> (sending again as don’t know if the stupid header ‘spoofed’ is
>> stopping
>> >> getting though – apologies if already sent)
>> >>
>> >> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
>> >>
>> >> Conrad,
>> >>
>> >> For s2s now you can just point at any of the nodes in the
>> cluster.
>> >> Have you tried changing the URL or removing and adding new
>> RPG
>> >> entries?
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
>> >> <co...@secdata.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I have finally taken the plunge to upgrade my cluster
>> from 0.6.1
>> >> to 1.0.0.
>> >> >
>> >> > 6 nodes with a NCM.
>> >> >
>> >> > With the removal of NCM in 1.0.0 I believe I now have an
>> issue
>> >> where none of
>> >> > my Remote Process Groups work as they previously did
>> because
>> >> they were
>> >> > configured to connect to the NCM (as the RPG url) which
>> now
>> >> doesn’t exist.
>> >> >
>> >> > I have tried converting my NCM to a node but whilst I can
>> get it
>> >> running
>> >> > (sort of) when I try and connect to the cluster I get
>> something
>> >> like this in
>> >> > my logs…
>> >> >
>> >> >
>> >> >
>> >> > 2016-10-19 13:14:44,109 ERROR [main]
>> >> o.a.nifi.controller.StandardFlowService
>> >> > Failed to load flow from cluster due to:
>> >> > org.apache.nifi.controller.UninheritableFlowException:
>> Failed to
>> >> connect
>> >> > node to cluster because local flow is different than
>> cluster
>> >> flow.
>> >> >
>> >> > org.apache.nifi.controller.UninheritableFlowException:
>> Failed to
>> >> connect
>> >> > node to cluster because local flow is different than
>> cluster
>> >> flow.
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> >> > [nifi-jetty-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > Caused by:
>> >> org.apache.nifi.controller.UninheritableFlowException: Proposed
>> >> > Authorizer is not inheritable by the flow controller
>> because of
>> >> Authorizer
>> >> > differences: Proposed Authorizations do not match current
>> >> Authorizations
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > ... 4 common frames omitted
>> >> >
>> >> > 2016-10-19 13:14:44,414 ERROR [main]
>> >> o.a.n.c.c.node.NodeClusterCoordinator
>> >> > Event Reported for ncm-cm1.local:9090 -- Node
>> disconnected from
>> >> > cluster due to
>> >> org.apache.nifi.controller.UninheritableFlowException: Failed
>> >> > to connect node to cluster because local flow is
>> different than
>> >> cluster
>> >> > flow.
>> >> >
>> >> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster
>> Coordinator]
>> >> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
>> >> Thread[Shutdown
>> >> > Cluster Coordinator,5,main]:
>> java.lang.NullPointerException
>> >> >
>> >> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster
>> Coordinator]
>> >> > org.apache.nifi.NiFi
>> >> >
>> >> > java.lang.NullPointerException: null
>> >> >
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
>> >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
>> >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at java.lang.Thread.run(Thread.java:745)
>> ~[na:1.8.0_51]
>> >> >
>> >> > 2016-10-19 13:14:44,448 WARN [main]
>> >> o.a.n.c.l.e.CuratorLeaderElectionManager
>> >> > Failed to close Leader Selector for Cluster Coordinator
>> >> >
>> >> > java.lang.IllegalStateException: Already closed or has
>> not been
>> >> started
>> >> >
>> >> > at
>> >> >
>> >>
>> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
>> >> > ~[guava-18.0.jar:na]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
>> >> > ~[curator-recipes-2.11.0.jar:na]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
>> >> > [nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
>> >> > [nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
>> >> > [nifi-jetty-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
>> >> > org.apache.nifi.io.socket.SocketListener Failed to
>> communicate
>> >> with Unknown
>> >> > Host due to java.net.SocketException: Socket closed
>> >> >
>> >> > java.net.SocketException: Socket closed
>> >> >
>> >> > at java.net.PlainSocketImpl.socketAccept(Native
>> Method)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
>> >> > ~[na:1.8.0_51]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
>> >> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>> >> >
>> >> > at java.lang.Thread.run(Thread.java:745)
>> [na:1.8.0_51]
>> >> >
>> >> > 2016-10-19 13:14:45,064 WARN [main]
>> >> org.apache.nifi.web.server.JettyServer
>> >> > Failed to start web server... shutting down.
>> >> >
>> >> > java.lang.Exception: Unable to load flow due to:
>> >> java.io.IOException:
>> >> > org.apache.nifi.controller.UninheritableFlowException:
>> Failed to
>> >> connect
>> >> > node to cluster because local flow is different than
>> cluster
>> >> flow.
>> >> >
>> >> > at
>> >> >
>> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
>> >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> >> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >> >
>> >> > Caused by: java.io.IOException:
>> >> > org.apache.nifi.controller.UninheritableFlowException:
>> Failed to
>> >> connect
>> >> > node to cluster because local flow is different than
>> cluster
>> >> flow.
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> >> >
>> >> > ... 2 common frames omitted
>> >> >
>> >> > Caused by:
>> >> org.apache.nifi.controller.UninheritableFlowException: Failed to
>> >> > connect node to cluster because local flow is different
>> than
>> >> cluster flow.
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > ... 3 common frames omitted
>> >> >
>> >> > Caused by:
>> >> org.apache.nifi.controller.UninheritableFlowException: Proposed
>> >> > Authorizer is not inheritable by the flow controller
>> because of
>> >> Authorizer
>> >> > differences: Proposed Authorizations do not match current
>> >> Authorizations
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >> >
>> >> > ... 4 common frames omitted
>> >> >
>> >> > [root@ncm-cm1 logs]#
>> >> >
>> >> >
>> >> >
>> >> > I don’t know if the ‘Proposed Authorizer is not
>> inheritable…’
>> >> exception is
>> >> > part of the problem too.
>> >> >
>> >> > The docs weren’t very clear on whether (when upgrading
>> and using
>> >> the legacy
>> >> > support of the authorized-user.xml path required the
>> nodes to be
>> >> also added
>> >> > to the authorizers.xml.
>> >> >
>> >> > I did add them in the end as various attempts to get the
>> cluster
>> >> up and
>> >> > running without them failed (as each server didn’t seem
>> to have
>> >> rights to do
>> >> > anything.
>> >> >
>> >> >
>> >> >
>> >> > I have a lot of RPG in my work flows as I am ingesting
>> many
>> >> syslog data
>> >> > sources and this was the recommended pattern to
>> distribute the
>> >> data
>> >> > (listensyslog…run on primary, output to port (RPG), pick
>> up in
>> >> rest of data
>> >> > flow),
>> >> >
>> >> >
>> >> >
>> >> > Any suggestions on where to start trying to get this
>> working?
>> >> >
>> >> > I’ve tried creating a new RPG on one on the nifis and
>> connecting
>> >> the
>> >> > syslog to that which sort of worked but then I have a
>> bunch of
>> >> other errors
>> >> > when trying to enable the ports to do with not being able
>> to
>> >> connect to
>> >> > (what was) the NCM.
>> >> >
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Conrad
>> >> >
>> >> >
>> >> >
>> >> > SecureData, combating cyber threats
>> >> >
>> >> > ________________________________
>> >> >
>> >> > The information contained in this message or any of its
>> >> attachments may be
>> >> > privileged and confidential and intended for the
>> exclusive use
>> >> of the
>> >> > intended recipient. If you are not the intended recipient
>> any
>> >> disclosure,
>> >> > reproduction, distribution or other dissemination or use
>> of this
>> >> > communications is strictly prohibited. The views
>> expressed in
>> >> this email are
>> >> > those of the individual and not necessarily of SecureData
>> Europe
>> >> Ltd. Any
>> >> > prices quoted are only valid if followed up by a formal
>> written
>> >> quote.
>> >> >
>> >> > SecureData Europe Limited. Registered in England & Wales
>> >> 04365896.
>> >> > Registered Address: SecureData House, Hermitage Court,
>> Hermitage
>> >> Lane,
>> >> > Maidstone, Kent, ME16 9NT
>> >>
>> >>
>> >> ***This email originated outside SecureData***
>> >>
>> >> Click
>> >>
>> https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==
>> >> to report this email as spam.
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > * Unknown Key
>> > * 0x2F7DEF69
>> >
>> >
>> >
>> >
>> >
>> > * Unknown Key
>> > * 0x2F7DEF69
>> >
>> >
>>
>>
>
>
>
> --
> Sent from Gmail Mobile
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Hi,
Yes, I think this is the issue.
I have changed the nifi.remote.input.socket.host to be nifi.remote.input.host.
I think I have over-thought the comment at the top of the admin docs for Site to Site
“Properties named with nifi.remote.input.socket.* are RAW transport protocol specific. Similarly, nifi.remote.input.http.* are HTTP transport protocol specific properties.
” and made the leap of ‘logic’ to include
a) Keep the ….input.socket.host from previous version
b) Include a …input.http.host
c) Include a nifi.remote.input.http.port
I don’t suppose b & c matter too much as they are most likely ignored, but not having the correct setting for a) probably doesn’t help.
With a vanilla install I still think not having http enables causes the NPE when using RAW.
If someone can confirm if there is an outstanding JIRA for this (or fixed on master branch) then I won’t record another one. If I do, probably will be a bit garbled given the probably root cause as above.
Thanks again for everyone’s input on this. I have now reverted back to my original 0.6.1 version then successfully upgraded to 1.0.0.
Conrad
From: Bryan Bende <bb...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Friday, 21 October 2016 at 16:09
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Conrad,
I'm glad you were able to get something working. I am definitely confused by the properties that were required though.
My understanding is that for RAW site-to-site it uses:
nifi.remote.input.host=
nifi.remote.input.socket.port=
And for http based site-to-site it uses the web http or https port that you have configured (same as the UI).
The properties you added seem to be slightly different. One thing I noticed was that nifi.remote.input.socket.host changed to nifi.remote.input.host in 1.0.0, so maybe that was a problem.
The latest base nifi.properties is here:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties
On Friday, October 21, 2016, Joe Witt <joe.witt@gmail.com<javascript:_e(%7B%7D,'cvml','joe.witt@gmail.com');>> wrote:
It's ok. If you see it again would love to get the stack trace. NPEs
automatically should turn into JIRAs as they should definitely never
happen. I'm not suggesting tracking this down will help your s2s/rpg
struggles but there is clearly something we can tighten up there.
On Fri, Oct 21, 2016 at 10:43 AM, Conrad Crampton
<co...@secdata.com> wrote:
> I’ve just tried to replicate but without success. Using a RPG with RAW transfer protocol.
>
> If you want I can try removing the additional lines in my nifi.properties (the ones for http.port and host that I added which appeared to get this working?
>
> Regards
> Conrad
>
>
> On 21/10/2016, 15:24, "Joe Witt" <jo...@gmail.com> wrote:
>
> Whoa there.... can you turn on debug logging in conf/logback.xml by adding
>
> <logger name="org.apache.nifi.remote.SocketRemoteSiteListener "
> level="DEBUG"/>
>
> Can you submit a JIRA for the log output with the stacktrace for those
> NPEs please.
>
> Thanks
> Joe
>
> On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
> <co...@secdata.com> wrote:
> > Hi,
> >
> > I realise this may be getting boring for most but…
> >
> > I didn’t find any resolution to the upgrade so I have cleanly installed
> > v1.0.0 and oddly experienced the same issue with RPGs in that although the
> > RPGs didn’t show any errors (in so much as they had no warning on them and
> > reported that S2S was secure) the errors reported were about not being able
> > to determine other nodes in cluster.
> >
> > This is a section of log that showed an error that I don’t think I saw
> > before but only including here in case one of you fine folks need it for any
> > reason…
> >
> >
> >
> > ERROR [Site-to-Site Worker Thread-194]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode3-cm1.mis.local:57289]
> > (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c8-bffc59d650a7])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode1-cm1.mis.local:35831]
> > (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c9-d294bb4037f5])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode5-cm1.mis.local:59093]
> > (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-8201-16b905e5beb6])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode6-cm1.mis.local:37275]
> > (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a149-4892d0193819])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode4-cm1.mis.local:57745]
> > (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-8047-b5ba4d9b67ec])
> > due to java.lang.NullPointerException; closing connection
> >
> >
> >
> > in my naivety this suggests a problem with the socket (RAW) connection
> > protocol. The relevant section for S2S connection in my nifi.properties is
> >
> > nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the number
> > different for each node obviously
> >
> > nifi.remote.input.socket.port=10443
> >
> > nifi.remote.input.secure=true
> >
> > nifi.remote.input.http.enabled=false
> >
> > nifi.remote.input.http.transaction.ttl=30 sec
> >
> >
> >
> > the errors suggest that the port specified here aren’t being used, but some
> > random ports are being used instead. Of course this is complete supposition
> > and probably a red herring.
> >
> >
> >
> > Anyway, I updated my nifi.properties to
> >
> > nifi.remote.input.socket.host=nifinode6-cm1.mis.local
> >
> > nifi.remote.input.http.host=nifinode6-cm1.mis.local
> >
> > nifi.remote.input.socket.port=10443
> >
> > nifi.remote.input.http.port=11443
> >
> > nifi.remote.input.secure=true
> >
> > nifi.remote.input.http.enabled=true
> >
> > nifi.remote.input.http.transaction.ttl=30 sec
> >
> >
> >
> > and used HTTP for my RPG and is now working ok.
> >
> >
> >
> > The test harness for this is
> >
> >
> >
> > GenerateFlowFile->RGP(test input port)
> >
> > InputPort(test input)->LogAttribute
> >
> >
> >
> > Regards,
> >
> > Conrad
> >
> >
> >
> >
> >
> >
> >
> > From: Conrad Crampton <co...@SecData.com>
> > Date: Friday, 21 October 2016 at 08:18
> >
> >
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> > Hi,
> >
> > Yes, the access policy for the input port at the root level of my workspace
> > has S2S access policy (receive data via site to site) for all server nodes
> > (I have all my nodes in one group).
> >
> > For the next level down in my workspace (I have other process groups from my
> > main ‘root’ page in the UI space to organise the flows), I have input ports
> > on the next level down of flows which when I check the access policies on
> > that for S2S, the receive and send data via site-to-site options are greyed
> > out and if I try to override the policy, they still are. I don’t know if
> > this is important, but from reading the post below, the issue that the
> > access policies looks to address is different from my issue. The RPG has a
> > locked padlock and says site to site is secure. I don’t have any warning
> > triangles on the RPG itself, but I have the aforementioned warnings in my
> > logs.
> >
> >
> >
> > I’m going to abandon this now as I can’t get it to work.
> >
> >
> >
> > I’m going to try a fresh install and go with that – and have to recreate all
> > my flows where necessary. I’m moving to a new model of data ingestion anyway
> > so isn’t as catastrophic as it might be.
> >
> >
> >
> > Thanks for the help.
> >
> > Conrad
> >
> >
> >
> > From: Andy LoPresto <al...@apache.org>
> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Date: Thursday, 20 October 2016 at 17:31
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> > * PGP Signed by an unknown key
> >
> > Conrad,
> >
> >
> >
> > For the site-to-site did you follow the instructions here [1]? Each node
> > needs to be added as a user in order to make the connections.
> >
> >
> >
> > [1]
> > http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
> >
> >
> >
> > Andy LoPresto
> >
> > alopresto@apache.org
> >
> > alopresto.apache@gmail.com
> >
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> >
> >
> > On Oct 20, 2016, at 7:36 AM, Conrad Crampton <co...@SecData.com>
> > wrote:
> >
> >
> >
> > Ok,
> >
> > So I tried everything suggested so far to no avail unfortunately.
> >
> >
> >
> > So what I have done is to create all new certs etc. using the tookit.
> > Updated my existing authoriszed-users.xml to have to match the full cert
> > distinguished names CN=server.name<http://server.name>, OU=NIFI etc.
> >
> >
> >
> > Recreated all my remote process groups to not reference the original NCM as
> > that still wouldn’t work – after a complete new install (upgrade).
> >
> >
> >
> > So now what I have is a six node cluster using original data/worker nodes
> > and they are part of the cluster – all appears to be working ie. I can log
> > into the UI (nice by the way ;-) on each server. There are no SSL handshake
> > errors etc. BUT the RPG (newly created) still don’t appear to be working. I
> > am getting
> >
> >
> >
> > 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> >
> > nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> >
> > nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> >
> > nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> >
> > nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> >
> > nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> >
> > nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> >
> > nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> >
> >
> > I can telnet from server to server on both https (UI) port and S2S port.
> >
> > I am really at a loss as to what to do now.
> >
> >
> >
> > Data is queuing up in my input processors with nowhere to go.
> >
> > Do I have to do something radical here to get this working like stopping
> > everything, clearing out all the queues then starting up again??? I really
> > don’t want to do this obviously but I am getting nowhere on this – two days
> > of frustration with nothing to show for it.
> >
> >
> >
> > Any more suggestions please??
> >
> > Thanks for your patience.
> >
> > Conrad
> >
> >
> >
> >
> >
> > From: Andy LoPresto <al...@apache.org>
> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Date: Wednesday, 19 October 2016 at 18:24
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> >> Old Signed by an unknown key
> >
> > Hi Conrad,
> >
> >
> >
> > Bryan is correct that changing the certificates (and the encapsulating
> > keystores and truststores) will not affect any data held in the nodes.
> >
> >
> >
> > Regenerating everything using the TLS toolkit should hopefully not be too
> > challenging, but I am also curious as to why you are getting these handshake
> > exceptions now. As Bryan pointed out, adding the following line to
> > bootstrap.conf will provide substantial additional log output which should
> > help trace the issue.
> >
> >
> >
> > java.arg.15=-Djavax.net.debug=ssl,handshake
> >
> >
> >
> > You can also imitate the node connecting to the (previous) NCM via this
> > command:
> >
> >
> >
> > $ openssl s_client -connect <host:port> -debug -state -cert
> > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> > <path_to_your_CA_cert.pem>
> >
> >
> >
> > Where:
> >
> >
> >
> > <host:port> = the hostname and port of the “NCM”
> > <path_to_your_cert.pem> = the public key used to identify the “node” (can be
> > exported from the node keystore [1])
> > <path_to_your_key.pem> = the private key used to identify the “node” (can be
> > exported from the node keystore via 2 step process)
> > <path_to_your_CA_cert.pem> = the public key used to sign the “NCM”
> > certificate (could be a 3rd party like Verisign or DigiCert, or an internal
> > organization CA if you have one)
> >
> >
> >
> > If you’ve already regenerated everything and it works, that’s fine. But if
> > you have the time to try and investigate the old certs, we are interested
> > and prepared to help. Thanks.
> >
> >
> >
> > [1] https://security.stackexchange.com/a/66865/16485
> >
> >
> >
> > Andy LoPresto
> >
> > alopresto@apache.org
> >
> > alopresto.apache@gmail.com
> >
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> >
> >
> > On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com> wrote:
> >
> >
> >
> > That is definitely weird that it is only an issue on the node that used to
> > be the NCM.
> >
> > Might be worth double checking the keystore and truststore of that one node
> > and make sure it has what you would expect, and also double check
> > nifi.properties compared to the others to see if anything seems different.
> >
> >
> >
> > Changing all of the keystores, truststores, etc should be fine from a data
> > perspective...
> >
> >
> >
> > If you decide to go that route it would probably be easiest to start back
> > over from a security perspective, meaning:
> >
> > - Stop all the nodes and delete the users.xml and authorizations.xml from
> > all nodes
> >
> > - Configure authorizers.xml with the appropriate initial admin (or legacy
> > file) and node identities based on the new certs
> >
> > - Ensure authorizers.xml is the same on all nodes
> >
> > - Then restart everything
> >
> >
> >
> > Alternatively, you might be able to manually add users for all of the new
> > certs before shutting everything down and give them the appropriate
> > policies, then restart everything, but requires more manual work to get
> > everything lined up.
> >
> >
> >
> >
> >
> > On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton
> > <co...@secdata.com> wrote:
> >
> > Hi,
> >
> > As a plan for tomorrow – I have generated new keystores, truststores, client
> > certts etc. for all nodes in my cluster using the
> >
> >
> >
> > From: Bryan Bende <bb...@gmail.com>
> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Date: Wednesday, 19 October 2016 at 15:33
> >
> >
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> >
> >
> > Trying to think of things to check here...
> >
> >
> >
> > Does every node have nifi.remote.input.secure=true in nifi.properties and
> > the URL in the RPG is an https URL?
> >
> >
> >
> > On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton
> > <co...@secdata.com> wrote:
> >
> > One other thing…
> >
> > The RPGs have an unlocked padlock on them saying S2S is not secure.
> >
> > Conrad
> >
> >
> >
> > From: Bryan Bende <bb...@gmail.com>
> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Date: Wednesday, 19 October 2016 at 15:20
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> > Ok that does seem like a TLS/SSL issue...
> >
> >
> >
> > Is this a single cluster doing site-to-site to itself?
> >
> >
> >
> > On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com> wrote:
> >
> > thanks conrad - did get it. Bryan is being more helpful that I so I
> > went silent :-)
> >
> > On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
> >
> > <co...@secdata.com> wrote:
> >> Hi Joe,
> >> Yep,
> >> Tried removing the RPG that referenced the NCM and adding new one with
> >> one of the nifis as url.
> >> That sort of worked, but kept getting errors about the NCM not being
> >> available for the ports and therefore couldn’t actually enable the port I
> >> needed to for that RPG.
> >> Thanks
> >> Conrad
> >>
> >> (sending again as don’t know if the stupid header ‘spoofed’ is stopping
> >> getting though – apologies if already sent)
> >>
> >> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
> >>
> >> Conrad,
> >>
> >> For s2s now you can just point at any of the nodes in the cluster.
> >> Have you tried changing the URL or removing and adding new RPG
> >> entries?
> >>
> >> Thanks
> >> Joe
> >>
> >> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> >> <co...@secdata.com> wrote:
> >> > Hi,
> >> >
> >> > I have finally taken the plunge to upgrade my cluster from 0.6.1
> >> to 1.0.0.
> >> >
> >> > 6 nodes with a NCM.
> >> >
> >> > With the removal of NCM in 1.0.0 I believe I now have an issue
> >> where none of
> >> > my Remote Process Groups work as they previously did because
> >> they were
> >> > configured to connect to the NCM (as the RPG url) which now
> >> doesn’t exist.
> >> >
> >> > I have tried converting my NCM to a node but whilst I can get it
> >> running
> >> > (sort of) when I try and connect to the cluster I get something
> >> like this in
> >> > my logs…
> >> >
> >> >
> >> >
> >> > 2016-10-19 13:14:44,109 ERROR [main]
> >> o.a.nifi.controller.StandardFlowService
> >> > Failed to load flow from cluster due to:
> >> > org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> connect
> >> > node to cluster because local flow is different than cluster
> >> flow.
> >> >
> >> > org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> connect
> >> > node to cluster because local flow is different than cluster
> >> flow.
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> >> > [nifi-jetty-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > Caused by:
> >> org.apache.nifi.controller.UninheritableFlowException: Proposed
> >> > Authorizer is not inheritable by the flow controller because of
> >> Authorizer
> >> > differences: Proposed Authorizations do not match current
> >> Authorizations
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.persistence.St<http://org.apache.nifi.persistence.St>andardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > ... 4 common frames omitted
> >> >
> >> > 2016-10-19 13:14:44,414 ERROR [main]
> >> o.a.n.c.c.node.NodeClusterCoordinator
> >> > Event Reported for ncm-cm1.local:9090 -- Node disconnected from
> >> > cluster due to
> >> org.apache.nifi.controller.UninheritableFlowException: Failed
> >> > to connect node to cluster because local flow is different than
> >> cluster
> >> > flow.
> >> >
> >> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> >> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
> >> Thread[Shutdown
> >> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >> >
> >> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> >> > org.apache.nifi.NiFi
> >> >
> >> > java.lang.NullPointerException: null
> >> >
> >> > at
> >> >
> >> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >> >
> >> > 2016-10-19 13:14:44,448 WARN [main]
> >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >> > Failed to close Leader Selector for Cluster Coordinator
> >> >
> >> > java.lang.IllegalStateException: Already closed or has not been
> >> started
> >> >
> >> > at
> >> >
> >> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> >> > ~[guava-18.0.jar:na]
> >> >
> >> > at
> >> >
> >> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> >> > ~[curator-recipes-2.11.0.jar:na]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> >> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> >> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> >> > [nifi-jetty-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> >> > org.apache.nifi.io.socket.SocketListener Failed to communicate
> >> with Unknown
> >> > Host due to java.net.SocketException: Socket closed
> >> >
> >> > java.net.SocketException: Socket closed
> >> >
> >> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> java.net<http://java.net>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> java.net.ServerSocket.implAccept(ServerSocket.java:545)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> >> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >> >
> >> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >> >
> >> > 2016-10-19 13:14:45,064 WARN [main]
> >> org.apache.nifi.web.server.JettyServer
> >> > Failed to start web server... shutting down.
> >> >
> >> > java.lang.Exception: Unable to load flow due to:
> >> java.io.IOException:
> >> > org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> connect
> >> > node to cluster because local flow is different than cluster
> >> flow.
> >> >
> >> > at
> >> >
> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > Caused by: java.io.IOException:
> >> > org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> connect
> >> > node to cluster because local flow is different than cluster
> >> flow.
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >> >
> >> > ... 2 common frames omitted
> >> >
> >> > Caused by:
> >> org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> > connect node to cluster because local flow is different than
> >> cluster flow.
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > ... 3 common frames omitted
> >> >
> >> > Caused by:
> >> org.apache.nifi.controller.UninheritableFlowException: Proposed
> >> > Authorizer is not inheritable by the flow controller because of
> >> Authorizer
> >> > differences: Proposed Authorizations do not match current
> >> Authorizations
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.persistence.St<http://org.apache.nifi.persistence.St>andardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > ... 4 common frames omitted
> >> >
> >> > [root@ncm-cm1 logs]#
> >> >
> >> >
> >> >
> >> > I don’t know if the ‘Proposed Authorizer is not inheritable…’
> >> exception is
> >> > part of the problem too.
> >> >
> >> > The docs weren’t very clear on whether (when upgrading and using
> >> the legacy
> >> > support of the authorized-user.xml path required the nodes to be
> >> also added
> >> > to the authorizers.xml.
> >> >
> >> > I did add them in the end as various attempts to get the cluster
> >> up and
> >> > running without them failed (as each server didn’t seem to have
> >> rights to do
> >> > anything.
> >> >
> >> >
> >> >
> >> > I have a lot of RPG in my work flows as I am ingesting many
> >> syslog data
> >> > sources and this was the recommended pattern to distribute the
> >> data
> >> > (listensyslog…run on primary, output to port (RPG), pick up in
> >> rest of data
> >> > flow),
> >> >
> >> >
> >> >
> >> > Any suggestions on where to start trying to get this working?
> >> >
> >> > I’ve tried creating a new RPG on one on the nifis and connecting
> >> the
> >> > syslog to that which sort of worked but then I have a bunch of
> >> other errors
> >> > when trying to enable the ports to do with not being able to
> >> connect to
> >> > (what was) the NCM.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Conrad
> >> >
> >> >
> >> >
> >> > SecureData, combating cyber threats
> >> >
> >> > ________________________________
> >> >
> >> > The information contained in this message or any of its
> >> attachments may be
> >> > privileged and confidential and intended for the exclusive use
> >> of the
> >> > intended recipient. If you are not the intended recipient any
> >> disclosure,
> >> > reproduction, distribution or other dissemination or use of this
> >> > communications is strictly prohibited. The views expressed in
> >> this email are
> >> > those of the individual and not necessarily of SecureData Europe
> >> Ltd. Any
> >> > prices quoted are only valid if followed up by a formal written
> >> quote.
> >> >
> >> > SecureData Europe Limited. Registered in England & Wales
> >> 04365896.
> >> > Registered Address: SecureData House, Hermitage Court, Hermitage
> >> Lane,
> >> > Maidstone, Kent, ME16 9NT
> >>
> >>
> >> ***This email originated outside SecureData***
> >>
> >> Click
> >> https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==>
> >> to report this email as spam.
> >>
> >>
> >>
> >>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > * Unknown Key
> > * 0x2F7DEF69
> >
> >
> >
> >
> >
> > * Unknown Key
> > * 0x2F7DEF69
> >
> >
>
>
--
Sent from Gmail Mobile
Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Bryan Bende <bb...@gmail.com>.
Conrad,
I'm glad you were able to get something working. I am definitely confused
by the properties that were required though.
My understanding is that for RAW site-to-site it uses:
nifi.remote.input.host=
nifi.remote.input.socket.port=
And for http based site-to-site it uses the web http or https port that you
have configured (same as the UI).
The properties you added seem to be slightly different. One thing I noticed
was that nifi.remote.input.socket.host changed to nifi.remote.input.host in
1.0.0, so maybe that was a problem.
The latest base nifi.properties is here:
https://github.com/apache/nifi/blob/master/nifi-nar-
bundles/nifi-framework-bundle/nifi-framework/nifi-resources/
src/main/resources/conf/nifi.properties
On Friday, October 21, 2016, Joe Witt <joe.witt@gmail.com
<javascript:_e(%7B%7D,'cvml','joe.witt@gmail.com');>> wrote:
> It's ok. If you see it again would love to get the stack trace. NPEs
> automatically should turn into JIRAs as they should definitely never
> happen. I'm not suggesting tracking this down will help your s2s/rpg
> struggles but there is clearly something we can tighten up there.
>
> On Fri, Oct 21, 2016 at 10:43 AM, Conrad Crampton
> <co...@secdata.com> wrote:
> > I’ve just tried to replicate but without success. Using a RPG with RAW
> transfer protocol.
> >
> > If you want I can try removing the additional lines in my
> nifi.properties (the ones for http.port and host that I added which
> appeared to get this working?
> >
> > Regards
> > Conrad
> >
> >
> > On 21/10/2016, 15:24, "Joe Witt" <jo...@gmail.com> wrote:
> >
> > Whoa there.... can you turn on debug logging in conf/logback.xml by
> adding
> >
> > <logger name="org.apache.nifi.remote.SocketRemoteSiteListener "
> > level="DEBUG"/>
> >
> > Can you submit a JIRA for the log output with the stacktrace for
> those
> > NPEs please.
> >
> > Thanks
> > Joe
> >
> > On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
> > <co...@secdata.com> wrote:
> > > Hi,
> > >
> > > I realise this may be getting boring for most but…
> > >
> > > I didn’t find any resolution to the upgrade so I have cleanly
> installed
> > > v1.0.0 and oddly experienced the same issue with RPGs in that
> although the
> > > RPGs didn’t show any errors (in so much as they had no warning on
> them and
> > > reported that S2S was secure) the errors reported were about not
> being able
> > > to determine other nodes in cluster.
> > >
> > > This is a section of log that showed an error that I don’t think I
> saw
> > > before but only including here in case one of you fine folks need
> it for any
> > > reason…
> > >
> > >
> > >
> > > ERROR [Site-to-Site Worker Thread-194]
> > > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate
> with remote
> > > instance Peer[url=nifi://nifinode3-cm1.mis.local:57289]
> > > (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c
> 8-bffc59d650a7])
> > > due to java.lang.NullPointerException; closing connection
> > >
> > > 2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195]
> > > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate
> with remote
> > > instance Peer[url=nifi://nifinode1-cm1.mis.local:35831]
> > > (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c
> 9-d294bb4037f5])
> > > due to java.lang.NullPointerException; closing connection
> > >
> > > 2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196]
> > > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate
> with remote
> > > instance Peer[url=nifi://nifinode5-cm1.mis.local:59093]
> > > (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-820
> 1-16b905e5beb6])
> > > due to java.lang.NullPointerException; closing connection
> > >
> > > 2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197]
> > > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate
> with remote
> > > instance Peer[url=nifi://nifinode6-cm1.mis.local:37275]
> > > (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a14
> 9-4892d0193819])
> > > due to java.lang.NullPointerException; closing connection
> > >
> > > 2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198]
> > > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate
> with remote
> > > instance Peer[url=nifi://nifinode4-cm1.mis.local:57745]
> > > (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-804
> 7-b5ba4d9b67ec])
> > > due to java.lang.NullPointerException; closing connection
> > >
> > >
> > >
> > > in my naivety this suggests a problem with the socket (RAW)
> connection
> > > protocol. The relevant section for S2S connection in my
> nifi.properties is
> > >
> > > nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the
> number
> > > different for each node obviously
> > >
> > > nifi.remote.input.socket.port=10443
> > >
> > > nifi.remote.input.secure=true
> > >
> > > nifi.remote.input.http.enabled=false
> > >
> > > nifi.remote.input.http.transaction.ttl=30 sec
> > >
> > >
> > >
> > > the errors suggest that the port specified here aren’t being used,
> but some
> > > random ports are being used instead. Of course this is complete
> supposition
> > > and probably a red herring.
> > >
> > >
> > >
> > > Anyway, I updated my nifi.properties to
> > >
> > > nifi.remote.input.socket.host=nifinode6-cm1.mis.local
> > >
> > > nifi.remote.input.http.host=nifinode6-cm1.mis.local
> > >
> > > nifi.remote.input.socket.port=10443
> > >
> > > nifi.remote.input.http.port=11443
> > >
> > > nifi.remote.input.secure=true
> > >
> > > nifi.remote.input.http.enabled=true
> > >
> > > nifi.remote.input.http.transaction.ttl=30 sec
> > >
> > >
> > >
> > > and used HTTP for my RPG and is now working ok.
> > >
> > >
> > >
> > > The test harness for this is
> > >
> > >
> > >
> > > GenerateFlowFile->RGP(test input port)
> > >
> > > InputPort(test input)->LogAttribute
> > >
> > >
> > >
> > > Regards,
> > >
> > > Conrad
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: Conrad Crampton <co...@SecData.com>
> > > Date: Friday, 21 October 2016 at 08:18
> > >
> > >
> > > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
> Groups
> > >
> > >
> > >
> > > Hi,
> > >
> > > Yes, the access policy for the input port at the root level of my
> workspace
> > > has S2S access policy (receive data via site to site) for all
> server nodes
> > > (I have all my nodes in one group).
> > >
> > > For the next level down in my workspace (I have other process
> groups from my
> > > main ‘root’ page in the UI space to organise the flows), I have
> input ports
> > > on the next level down of flows which when I check the access
> policies on
> > > that for S2S, the receive and send data via site-to-site options
> are greyed
> > > out and if I try to override the policy, they still are. I don’t
> know if
> > > this is important, but from reading the post below, the issue that
> the
> > > access policies looks to address is different from my issue. The
> RPG has a
> > > locked padlock and says site to site is secure. I don’t have any
> warning
> > > triangles on the RPG itself, but I have the aforementioned
> warnings in my
> > > logs.
> > >
> > >
> > >
> > > I’m going to abandon this now as I can’t get it to work.
> > >
> > >
> > >
> > > I’m going to try a fresh install and go with that – and have to
> recreate all
> > > my flows where necessary. I’m moving to a new model of data
> ingestion anyway
> > > so isn’t as catastrophic as it might be.
> > >
> > >
> > >
> > > Thanks for the help.
> > >
> > > Conrad
> > >
> > >
> > >
> > > From: Andy LoPresto <al...@apache.org>
> > > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > > Date: Thursday, 20 October 2016 at 17:31
> > > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
> Groups
> > >
> > >
> > >
> > > * PGP Signed by an unknown key
> > >
> > > Conrad,
> > >
> > >
> > >
> > > For the site-to-site did you follow the instructions here [1]?
> Each node
> > > needs to be added as a user in order to make the connections.
> > >
> > >
> > >
> > > [1]
> > > http://bryanbende.com/development/2016/08/30/apache-nifi-1.
> 0.0-secure-site-to-site
> > >
> > >
> > >
> > > Andy LoPresto
> > >
> > > alopresto@apache.org
> > >
> > > alopresto.apache@gmail.com
> > >
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> > >
> > >
> > > On Oct 20, 2016, at 7:36 AM, Conrad Crampton
> <co...@SecData.com>
> > > wrote:
> > >
> > >
> > >
> > > Ok,
> > >
> > > So I tried everything suggested so far to no avail unfortunately.
> > >
> > >
> > >
> > > So what I have done is to create all new certs etc. using the
> tookit.
> > > Updated my existing authoriszed-users.xml to have to match the
> full cert
> > > distinguished names CN=server.name, OU=NIFI etc.
> > >
> > >
> > >
> > > Recreated all my remote process groups to not reference the
> original NCM as
> > > that still wouldn’t work – after a complete new install (upgrade).
> > >
> > >
> > >
> > > So now what I have is a six node cluster using original
> data/worker nodes
> > > and they are part of the cluster – all appears to be working ie. I
> can log
> > > into the UI (nice by the way ;-) on each server. There are no SSL
> handshake
> > > errors etc. BUT the RPG (newly created) still don’t appear to be
> working. I
> > > am getting
> > >
> > >
> > >
> > > 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> > >
> > > nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelect
> or@782af623
> > > Unable to refresh Remote Group's peers due to Unable to
> communicate with
> > > remote NiFi cluster in order to determine which nodes exist in the
> remote
> > > cluster
> > >
> > > 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> > >
> > > nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelect
> or@3a547274
> > > Unable to refresh Remote Group's peers due to Unable to
> communicate with
> > > remote NiFi cluster in order to determine which nodes exist in the
> remote
> > > cluster
> > >
> > > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> > >
> > > nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelect
> or@54c2df1
> > > Unable to refresh Remote Group's peers due to Unable to
> communicate with
> > > remote NiFi cluster in order to determine which nodes exist in the
> remote
> > > cluster
> > >
> > > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> > >
> > > nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelect
> or@50d59f3c
> > > Unable to refresh Remote Group's peers due to Unable to
> communicate with
> > > remote NiFi cluster in order to determine which nodes exist in the
> remote
> > > cluster
> > >
> > > 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> > >
> > > nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelect
> or@97c92ef
> > > Unable to refresh Remote Group's peers due to Unable to
> communicate with
> > > remote NiFi cluster in order to determine which nodes exist in the
> remote
> > > cluster
> > >
> > > 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> > >
> > > nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelect
> or@70663037
> > > Unable to refresh Remote Group's peers due to Unable to
> communicate with
> > > remote NiFi cluster in order to determine which nodes exist in the
> remote
> > > cluster
> > >
> > > 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> > >
> > > nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelect
> or@3c040426
> > > Unable to refresh Remote Group's peers due to Unable to
> communicate with
> > > remote NiFi cluster in order to determine which nodes exist in the
> remote
> > > cluster
> > >
> > >
> > >
> > > I can telnet from server to server on both https (UI) port and S2S
> port.
> > >
> > > I am really at a loss as to what to do now.
> > >
> > >
> > >
> > > Data is queuing up in my input processors with nowhere to go.
> > >
> > > Do I have to do something radical here to get this working like
> stopping
> > > everything, clearing out all the queues then starting up again???
> I really
> > > don’t want to do this obviously but I am getting nowhere on this –
> two days
> > > of frustration with nothing to show for it.
> > >
> > >
> > >
> > > Any more suggestions please??
> > >
> > > Thanks for your patience.
> > >
> > > Conrad
> > >
> > >
> > >
> > >
> > >
> > > From: Andy LoPresto <al...@apache.org>
> > > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > > Date: Wednesday, 19 October 2016 at 18:24
> > > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
> Groups
> > >
> > >
> > >
> > >> Old Signed by an unknown key
> > >
> > > Hi Conrad,
> > >
> > >
> > >
> > > Bryan is correct that changing the certificates (and the
> encapsulating
> > > keystores and truststores) will not affect any data held in the
> nodes.
> > >
> > >
> > >
> > > Regenerating everything using the TLS toolkit should hopefully not
> be too
> > > challenging, but I am also curious as to why you are getting these
> handshake
> > > exceptions now. As Bryan pointed out, adding the following line to
> > > bootstrap.conf will provide substantial additional log output
> which should
> > > help trace the issue.
> > >
> > >
> > >
> > > java.arg.15=-Djavax.net.debug=ssl,handshake
> > >
> > >
> > >
> > > You can also imitate the node connecting to the (previous) NCM via
> this
> > > command:
> > >
> > >
> > >
> > > $ openssl s_client -connect <host:port> -debug -state -cert
> > > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> > > <path_to_your_CA_cert.pem>
> > >
> > >
> > >
> > > Where:
> > >
> > >
> > >
> > > <host:port> = the hostname and port of the “NCM”
> > > <path_to_your_cert.pem> = the public key used to identify the
> “node” (can be
> > > exported from the node keystore [1])
> > > <path_to_your_key.pem> = the private key used to identify the
> “node” (can be
> > > exported from the node keystore via 2 step process)
> > > <path_to_your_CA_cert.pem> = the public key used to sign the “NCM”
> > > certificate (could be a 3rd party like Verisign or DigiCert, or an
> internal
> > > organization CA if you have one)
> > >
> > >
> > >
> > > If you’ve already regenerated everything and it works, that’s
> fine. But if
> > > you have the time to try and investigate the old certs, we are
> interested
> > > and prepared to help. Thanks.
> > >
> > >
> > >
> > > [1] https://security.stackexchange.com/a/66865/16485
> > >
> > >
> > >
> > > Andy LoPresto
> > >
> > > alopresto@apache.org
> > >
> > > alopresto.apache@gmail.com
> > >
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> > >
> > >
> > > On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com> wrote:
> > >
> > >
> > >
> > > That is definitely weird that it is only an issue on the node that
> used to
> > > be the NCM.
> > >
> > > Might be worth double checking the keystore and truststore of that
> one node
> > > and make sure it has what you would expect, and also double check
> > > nifi.properties compared to the others to see if anything seems
> different.
> > >
> > >
> > >
> > > Changing all of the keystores, truststores, etc should be fine
> from a data
> > > perspective...
> > >
> > >
> > >
> > > If you decide to go that route it would probably be easiest to
> start back
> > > over from a security perspective, meaning:
> > >
> > > - Stop all the nodes and delete the users.xml and
> authorizations.xml from
> > > all nodes
> > >
> > > - Configure authorizers.xml with the appropriate initial admin (or
> legacy
> > > file) and node identities based on the new certs
> > >
> > > - Ensure authorizers.xml is the same on all nodes
> > >
> > > - Then restart everything
> > >
> > >
> > >
> > > Alternatively, you might be able to manually add users for all of
> the new
> > > certs before shutting everything down and give them the appropriate
> > > policies, then restart everything, but requires more manual work
> to get
> > > everything lined up.
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton
> > > <co...@secdata.com> wrote:
> > >
> > > Hi,
> > >
> > > As a plan for tomorrow – I have generated new keystores,
> truststores, client
> > > certts etc. for all nodes in my cluster using the
> > >
> > >
> > >
> > > From: Bryan Bende <bb...@gmail.com>
> > > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > > Date: Wednesday, 19 October 2016 at 15:33
> > >
> > >
> > > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
> Groups
> > >
> > >
> > >
> > >
> > >
> > > Trying to think of things to check here...
> > >
> > >
> > >
> > > Does every node have nifi.remote.input.secure=true in
> nifi.properties and
> > > the URL in the RPG is an https URL?
> > >
> > >
> > >
> > > On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton
> > > <co...@secdata.com> wrote:
> > >
> > > One other thing…
> > >
> > > The RPGs have an unlocked padlock on them saying S2S is not secure.
> > >
> > > Conrad
> > >
> > >
> > >
> > > From: Bryan Bende <bb...@gmail.com>
> > > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > > Date: Wednesday, 19 October 2016 at 15:20
> > > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process
> Groups
> > >
> > >
> > >
> > > Ok that does seem like a TLS/SSL issue...
> > >
> > >
> > >
> > > Is this a single cluster doing site-to-site to itself?
> > >
> > >
> > >
> > > On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>
> wrote:
> > >
> > > thanks conrad - did get it. Bryan is being more helpful that I so
> I
> > > went silent :-)
> > >
> > > On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
> > >
> > > <co...@secdata.com> wrote:
> > >> Hi Joe,
> > >> Yep,
> > >> Tried removing the RPG that referenced the NCM and adding new
> one with
> > >> one of the nifis as url.
> > >> That sort of worked, but kept getting errors about the NCM
> not being
> > >> available for the ports and therefore couldn’t actually enable
> the port I
> > >> needed to for that RPG.
> > >> Thanks
> > >> Conrad
> > >>
> > >> (sending again as don’t know if the stupid header ‘spoofed’ is
> stopping
> > >> getting though – apologies if already sent)
> > >>
> > >> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
> > >>
> > >> Conrad,
> > >>
> > >> For s2s now you can just point at any of the nodes in the
> cluster.
> > >> Have you tried changing the URL or removing and adding
> new RPG
> > >> entries?
> > >>
> > >> Thanks
> > >> Joe
> > >>
> > >> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> > >> <co...@secdata.com> wrote:
> > >> > Hi,
> > >> >
> > >> > I have finally taken the plunge to upgrade my cluster
> from 0.6.1
> > >> to 1.0.0.
> > >> >
> > >> > 6 nodes with a NCM.
> > >> >
> > >> > With the removal of NCM in 1.0.0 I believe I now have
> an issue
> > >> where none of
> > >> > my Remote Process Groups work as they previously did
> because
> > >> they were
> > >> > configured to connect to the NCM (as the RPG url) which
> now
> > >> doesn’t exist.
> > >> >
> > >> > I have tried converting my NCM to a node but whilst I
> can get it
> > >> running
> > >> > (sort of) when I try and connect to the cluster I get
> something
> > >> like this in
> > >> > my logs…
> > >> >
> > >> >
> > >> >
> > >> > 2016-10-19 13:14:44,109 ERROR [main]
> > >> o.a.nifi.controller.StandardFlowService
> > >> > Failed to load flow from cluster due to:
> > >> > org.apache.nifi.controller.UninheritableFlowException:
> Failed to
> > >> connect
> > >> > node to cluster because local flow is different than
> cluster
> > >> flow.
> > >> >
> > >> > org.apache.nifi.controller.UninheritableFlowException:
> Failed to
> > >> connect
> > >> > node to cluster because local flow is different than
> cluster
> > >> flow.
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.loadFromConne
> ctionResponse(StandardFlowService.java:879)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.load(Standard
> FlowService.java:493)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.web.server.JettyServer.start(JettyServer.jav
> a:746)
> > >> > [nifi-jetty-1.0.0.jar:1.0.0]
> > >> >
> > >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > >> > [nifi-runtime-1.0.0.jar:1.0.0]
> > >> >
> > >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > >> > [nifi-runtime-1.0.0.jar:1.0.0]
> > >> >
> > >> > Caused by:
> > >> org.apache.nifi.controller.UninheritableFlowException: Proposed
> > >> > Authorizer is not inheritable by the flow controller
> because of
> > >> Authorizer
> > >> > differences: Proposed Authorizations do not match
> current
> > >> Authorizations
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowSynchronizer.sync(Sta
> ndardFlowSynchronizer.java:252)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.FlowController.synchronize(FlowCo
> ntroller.java:1435)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.
> load(StandardXMLFlowConfigurationDAO.java:83)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.loadFromBytes
> (StandardFlowService.java:671)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.loadFromConne
> ctionResponse(StandardFlowService.java:857)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > ... 4 common frames omitted
> > >> >
> > >> > 2016-10-19 13:14:44,414 ERROR [main]
> > >> o.a.n.c.c.node.NodeClusterCoordinator
> > >> > Event Reported for ncm-cm1.local:9090 -- Node
> disconnected from
> > >> > cluster due to
> > >> org.apache.nifi.controller.UninheritableFlowException: Failed
> > >> > to connect node to cluster because local flow is
> different than
> > >> cluster
> > >> > flow.
> > >> >
> > >> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster
> Coordinator]
> > >> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
> > >> Thread[Shutdown
> > >> > Cluster Coordinator,5,main]:
> java.lang.NullPointerException
> > >> >
> > >> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster
> Coordinator]
> > >> > org.apache.nifi.NiFi
> > >> >
> > >> > java.lang.NullPointerException: null
> > >> >
> > >> > at
> > >> >
> > >> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHash
> Map.java:1011)
> > >> > ~[na:1.8.0_51]
> > >> >
> > >> > at
> > >> >
> > >> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap
> .java:1006)
> > >> > ~[na:1.8.0_51]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.cluster.coordination.node.NodeClusterCoordin
> ator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.cluster.coordination.node.NodeClusterCoordin
> ator.shutdown(NodeClusterCoordinator.java:119)
> > >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService$1.run(Standar
> dFlowService.java:330)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at java.lang.Thread.run(Thread.java:745)
> ~[na:1.8.0_51]
> > >> >
> > >> > 2016-10-19 13:14:44,448 WARN [main]
> > >> o.a.n.c.l.e.CuratorLeaderElectionManager
> > >> > Failed to close Leader Selector for Cluster Coordinator
> > >> >
> > >> > java.lang.IllegalStateException: Already closed or has
> not been
> > >> started
> > >> >
> > >> > at
> > >> >
> > >> com.google.common.base.Preconditions.checkState(Precondition
> s.java:173)
> > >> > ~[guava-18.0.jar:na]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.curator.framework.recipes.leader.LeaderSelector.c
> lose(LeaderSelector.java:270)
> > >> > ~[curator-recipes-2.11.0.jar:na]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.leader.election.CuratorLeaderElec
> tionManager.stop(CuratorLeaderElectionManager.java:159)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.FlowController.shutdown(FlowContr
> oller.java:1303)
> > >> > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.stop(Standard
> FlowService.java:339)
> > >> > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.web.server.JettyServer.start(JettyServer.jav
> a:753)
> > >> > [nifi-jetty-1.0.0.jar:1.0.0]
> > >> >
> > >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > >> > [nifi-runtime-1.0.0.jar:1.0.0]
> > >> >
> > >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > >> > [nifi-runtime-1.0.0.jar:1.0.0]
> > >> >
> > >> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > >> > org.apache.nifi.io.socket.SocketListener Failed to
> communicate
> > >> with Unknown
> > >> > Host due to java.net.SocketException: Socket closed
> > >> >
> > >> > java.net.SocketException: Socket closed
> > >> >
> > >> > at java.net.PlainSocketImpl.socketAccept(Native
> Method)
> > >> > ~[na:1.8.0_51]
> > >> >
> > >> > at
> > >> >
> > >> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketI
> mpl.java:404)
> > >> > ~[na:1.8.0_51]
> > >> >
> > >> > at
> > >> java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > >> > ~[na:1.8.0_51]
> > >> >
> > >> > at
> > >> >
> > >> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketI
> mpl.java:348)
> > >> > ~[na:1.8.0_51]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.io.socket.SocketListener$2.run(SocketListene
> r.java:112)
> > >> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> > >> >
> > >> > at java.lang.Thread.run(Thread.java:745)
> [na:1.8.0_51]
> > >> >
> > >> > 2016-10-19 13:14:45,064 WARN [main]
> > >> org.apache.nifi.web.server.JettyServer
> > >> > Failed to start web server... shutting down.
> > >> >
> > >> > java.lang.Exception: Unable to load flow due to:
> > >> java.io.IOException:
> > >> > org.apache.nifi.controller.UninheritableFlowException:
> Failed to
> > >> connect
> > >> > node to cluster because local flow is different than
> cluster
> > >> flow.
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.web.server.JettyServer.start(JettyServer.jav
> a:755)
> > >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >> >
> > >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > >> > [nifi-runtime-1.0.0.jar:1.0.0]
> > >> >
> > >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > >> > [nifi-runtime-1.0.0.jar:1.0.0]
> > >> >
> > >> > Caused by: java.io.IOException:
> > >> > org.apache.nifi.controller.UninheritableFlowException:
> Failed to
> > >> connect
> > >> > node to cluster because local flow is different than
> cluster
> > >> flow.
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.load(Standard
> FlowService.java:497)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.web.server.JettyServer.start(JettyServer.jav
> a:746)
> > >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >> >
> > >> > ... 2 common frames omitted
> > >> >
> > >> > Caused by:
> > >> org.apache.nifi.controller.UninheritableFlowException: Failed to
> > >> > connect node to cluster because local flow is different
> than
> > >> cluster flow.
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.loadFromConne
> ctionResponse(StandardFlowService.java:879)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.load(Standard
> FlowService.java:493)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > ... 3 common frames omitted
> > >> >
> > >> > Caused by:
> > >> org.apache.nifi.controller.UninheritableFlowException: Proposed
> > >> > Authorizer is not inheritable by the flow controller
> because of
> > >> Authorizer
> > >> > differences: Proposed Authorizations do not match
> current
> > >> Authorizations
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowSynchronizer.sync(Sta
> ndardFlowSynchronizer.java:252)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.FlowController.synchronize(FlowCo
> ntroller.java:1435)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.
> load(StandardXMLFlowConfigurationDAO.java:83)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.loadFromBytes
> (StandardFlowService.java:671)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > at
> > >> >
> > >> org.apache.nifi.controller.StandardFlowService.loadFromConne
> ctionResponse(StandardFlowService.java:857)
> > >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >> >
> > >> > ... 4 common frames omitted
> > >> >
> > >> > [root@ncm-cm1 logs]#
> > >> >
> > >> >
> > >> >
> > >> > I don’t know if the ‘Proposed Authorizer is not
> inheritable…’
> > >> exception is
> > >> > part of the problem too.
> > >> >
> > >> > The docs weren’t very clear on whether (when upgrading
> and using
> > >> the legacy
> > >> > support of the authorized-user.xml path required the
> nodes to be
> > >> also added
> > >> > to the authorizers.xml.
> > >> >
> > >> > I did add them in the end as various attempts to get
> the cluster
> > >> up and
> > >> > running without them failed (as each server didn’t seem
> to have
> > >> rights to do
> > >> > anything.
> > >> >
> > >> >
> > >> >
> > >> > I have a lot of RPG in my work flows as I am ingesting
> many
> > >> syslog data
> > >> > sources and this was the recommended pattern to
> distribute the
> > >> data
> > >> > (listensyslog…run on primary, output to port (RPG),
> pick up in
> > >> rest of data
> > >> > flow),
> > >> >
> > >> >
> > >> >
> > >> > Any suggestions on where to start trying to get this
> working?
> > >> >
> > >> > I’ve tried creating a new RPG on one on the nifis and
> connecting
> > >> the
> > >> > syslog to that which sort of worked but then I have a
> bunch of
> > >> other errors
> > >> > when trying to enable the ports to do with not being
> able to
> > >> connect to
> > >> > (what was) the NCM.
> > >> >
> > >> >
> > >> >
> > >> > Thanks
> > >> >
> > >> > Conrad
> > >> >
> > >> >
> > >> >
> > >> > SecureData, combating cyber threats
> > >> >
> > >> > ________________________________
> > >> >
> > >> > The information contained in this message or any of its
> > >> attachments may be
> > >> > privileged and confidential and intended for the
> exclusive use
> > >> of the
> > >> > intended recipient. If you are not the intended
> recipient any
> > >> disclosure,
> > >> > reproduction, distribution or other dissemination or
> use of this
> > >> > communications is strictly prohibited. The views
> expressed in
> > >> this email are
> > >> > those of the individual and not necessarily of
> SecureData Europe
> > >> Ltd. Any
> > >> > prices quoted are only valid if followed up by a formal
> written
> > >> quote.
> > >> >
> > >> > SecureData Europe Limited. Registered in England & Wales
> > >> 04365896.
> > >> > Registered Address: SecureData House, Hermitage Court,
> Hermitage
> > >> Lane,
> > >> > Maidstone, Kent, ME16 9NT
> > >>
> > >>
> > >> ***This email originated outside SecureData***
> > >>
> > >> Click
> > >> https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1
> Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==
> > >> to report this email as spam.
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > * Unknown Key
> > > * 0x2F7DEF69
> > >
> > >
> > >
> > >
> > >
> > > * Unknown Key
> > > * 0x2F7DEF69
> > >
> > >
> >
> >
>
>
--
Sent from Gmail Mobile
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Joe Witt <jo...@gmail.com>.
It's ok. If you see it again would love to get the stack trace. NPEs
automatically should turn into JIRAs as they should definitely never
happen. I'm not suggesting tracking this down will help your s2s/rpg
struggles but there is clearly something we can tighten up there.
On Fri, Oct 21, 2016 at 10:43 AM, Conrad Crampton
<co...@secdata.com> wrote:
> I’ve just tried to replicate but without success. Using a RPG with RAW transfer protocol.
>
> If you want I can try removing the additional lines in my nifi.properties (the ones for http.port and host that I added which appeared to get this working?
>
> Regards
> Conrad
>
>
> On 21/10/2016, 15:24, "Joe Witt" <jo...@gmail.com> wrote:
>
> Whoa there.... can you turn on debug logging in conf/logback.xml by adding
>
> <logger name="org.apache.nifi.remote.SocketRemoteSiteListener "
> level="DEBUG"/>
>
> Can you submit a JIRA for the log output with the stacktrace for those
> NPEs please.
>
> Thanks
> Joe
>
> On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
> <co...@secdata.com> wrote:
> > Hi,
> >
> > I realise this may be getting boring for most but…
> >
> > I didn’t find any resolution to the upgrade so I have cleanly installed
> > v1.0.0 and oddly experienced the same issue with RPGs in that although the
> > RPGs didn’t show any errors (in so much as they had no warning on them and
> > reported that S2S was secure) the errors reported were about not being able
> > to determine other nodes in cluster.
> >
> > This is a section of log that showed an error that I don’t think I saw
> > before but only including here in case one of you fine folks need it for any
> > reason…
> >
> >
> >
> > ERROR [Site-to-Site Worker Thread-194]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode3-cm1.mis.local:57289]
> > (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c8-bffc59d650a7])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode1-cm1.mis.local:35831]
> > (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c9-d294bb4037f5])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode5-cm1.mis.local:59093]
> > (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-8201-16b905e5beb6])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode6-cm1.mis.local:37275]
> > (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a149-4892d0193819])
> > due to java.lang.NullPointerException; closing connection
> >
> > 2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198]
> > o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> > instance Peer[url=nifi://nifinode4-cm1.mis.local:57745]
> > (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-8047-b5ba4d9b67ec])
> > due to java.lang.NullPointerException; closing connection
> >
> >
> >
> > in my naivety this suggests a problem with the socket (RAW) connection
> > protocol. The relevant section for S2S connection in my nifi.properties is
> >
> > nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the number
> > different for each node obviously
> >
> > nifi.remote.input.socket.port=10443
> >
> > nifi.remote.input.secure=true
> >
> > nifi.remote.input.http.enabled=false
> >
> > nifi.remote.input.http.transaction.ttl=30 sec
> >
> >
> >
> > the errors suggest that the port specified here aren’t being used, but some
> > random ports are being used instead. Of course this is complete supposition
> > and probably a red herring.
> >
> >
> >
> > Anyway, I updated my nifi.properties to
> >
> > nifi.remote.input.socket.host=nifinode6-cm1.mis.local
> >
> > nifi.remote.input.http.host=nifinode6-cm1.mis.local
> >
> > nifi.remote.input.socket.port=10443
> >
> > nifi.remote.input.http.port=11443
> >
> > nifi.remote.input.secure=true
> >
> > nifi.remote.input.http.enabled=true
> >
> > nifi.remote.input.http.transaction.ttl=30 sec
> >
> >
> >
> > and used HTTP for my RPG and is now working ok.
> >
> >
> >
> > The test harness for this is
> >
> >
> >
> > GenerateFlowFile->RGP(test input port)
> >
> > InputPort(test input)->LogAttribute
> >
> >
> >
> > Regards,
> >
> > Conrad
> >
> >
> >
> >
> >
> >
> >
> > From: Conrad Crampton <co...@SecData.com>
> > Date: Friday, 21 October 2016 at 08:18
> >
> >
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> > Hi,
> >
> > Yes, the access policy for the input port at the root level of my workspace
> > has S2S access policy (receive data via site to site) for all server nodes
> > (I have all my nodes in one group).
> >
> > For the next level down in my workspace (I have other process groups from my
> > main ‘root’ page in the UI space to organise the flows), I have input ports
> > on the next level down of flows which when I check the access policies on
> > that for S2S, the receive and send data via site-to-site options are greyed
> > out and if I try to override the policy, they still are. I don’t know if
> > this is important, but from reading the post below, the issue that the
> > access policies looks to address is different from my issue. The RPG has a
> > locked padlock and says site to site is secure. I don’t have any warning
> > triangles on the RPG itself, but I have the aforementioned warnings in my
> > logs.
> >
> >
> >
> > I’m going to abandon this now as I can’t get it to work.
> >
> >
> >
> > I’m going to try a fresh install and go with that – and have to recreate all
> > my flows where necessary. I’m moving to a new model of data ingestion anyway
> > so isn’t as catastrophic as it might be.
> >
> >
> >
> > Thanks for the help.
> >
> > Conrad
> >
> >
> >
> > From: Andy LoPresto <al...@apache.org>
> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Date: Thursday, 20 October 2016 at 17:31
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> > * PGP Signed by an unknown key
> >
> > Conrad,
> >
> >
> >
> > For the site-to-site did you follow the instructions here [1]? Each node
> > needs to be added as a user in order to make the connections.
> >
> >
> >
> > [1]
> > http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
> >
> >
> >
> > Andy LoPresto
> >
> > alopresto@apache.org
> >
> > alopresto.apache@gmail.com
> >
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> >
> >
> > On Oct 20, 2016, at 7:36 AM, Conrad Crampton <co...@SecData.com>
> > wrote:
> >
> >
> >
> > Ok,
> >
> > So I tried everything suggested so far to no avail unfortunately.
> >
> >
> >
> > So what I have done is to create all new certs etc. using the tookit.
> > Updated my existing authoriszed-users.xml to have to match the full cert
> > distinguished names CN=server.name, OU=NIFI etc.
> >
> >
> >
> > Recreated all my remote process groups to not reference the original NCM as
> > that still wouldn’t work – after a complete new install (upgrade).
> >
> >
> >
> > So now what I have is a six node cluster using original data/worker nodes
> > and they are part of the cluster – all appears to be working ie. I can log
> > into the UI (nice by the way ;-) on each server. There are no SSL handshake
> > errors etc. BUT the RPG (newly created) still don’t appear to be working. I
> > am getting
> >
> >
> >
> > 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> >
> > nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> >
> > nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> >
> > nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> >
> > nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> >
> > nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> >
> > nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> > 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> >
> > nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426
> > Unable to refresh Remote Group's peers due to Unable to communicate with
> > remote NiFi cluster in order to determine which nodes exist in the remote
> > cluster
> >
> >
> >
> > I can telnet from server to server on both https (UI) port and S2S port.
> >
> > I am really at a loss as to what to do now.
> >
> >
> >
> > Data is queuing up in my input processors with nowhere to go.
> >
> > Do I have to do something radical here to get this working like stopping
> > everything, clearing out all the queues then starting up again??? I really
> > don’t want to do this obviously but I am getting nowhere on this – two days
> > of frustration with nothing to show for it.
> >
> >
> >
> > Any more suggestions please??
> >
> > Thanks for your patience.
> >
> > Conrad
> >
> >
> >
> >
> >
> > From: Andy LoPresto <al...@apache.org>
> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Date: Wednesday, 19 October 2016 at 18:24
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> >> Old Signed by an unknown key
> >
> > Hi Conrad,
> >
> >
> >
> > Bryan is correct that changing the certificates (and the encapsulating
> > keystores and truststores) will not affect any data held in the nodes.
> >
> >
> >
> > Regenerating everything using the TLS toolkit should hopefully not be too
> > challenging, but I am also curious as to why you are getting these handshake
> > exceptions now. As Bryan pointed out, adding the following line to
> > bootstrap.conf will provide substantial additional log output which should
> > help trace the issue.
> >
> >
> >
> > java.arg.15=-Djavax.net.debug=ssl,handshake
> >
> >
> >
> > You can also imitate the node connecting to the (previous) NCM via this
> > command:
> >
> >
> >
> > $ openssl s_client -connect <host:port> -debug -state -cert
> > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> > <path_to_your_CA_cert.pem>
> >
> >
> >
> > Where:
> >
> >
> >
> > <host:port> = the hostname and port of the “NCM”
> > <path_to_your_cert.pem> = the public key used to identify the “node” (can be
> > exported from the node keystore [1])
> > <path_to_your_key.pem> = the private key used to identify the “node” (can be
> > exported from the node keystore via 2 step process)
> > <path_to_your_CA_cert.pem> = the public key used to sign the “NCM”
> > certificate (could be a 3rd party like Verisign or DigiCert, or an internal
> > organization CA if you have one)
> >
> >
> >
> > If you’ve already regenerated everything and it works, that’s fine. But if
> > you have the time to try and investigate the old certs, we are interested
> > and prepared to help. Thanks.
> >
> >
> >
> > [1] https://security.stackexchange.com/a/66865/16485
> >
> >
> >
> > Andy LoPresto
> >
> > alopresto@apache.org
> >
> > alopresto.apache@gmail.com
> >
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> >
> >
> > On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com> wrote:
> >
> >
> >
> > That is definitely weird that it is only an issue on the node that used to
> > be the NCM.
> >
> > Might be worth double checking the keystore and truststore of that one node
> > and make sure it has what you would expect, and also double check
> > nifi.properties compared to the others to see if anything seems different.
> >
> >
> >
> > Changing all of the keystores, truststores, etc should be fine from a data
> > perspective...
> >
> >
> >
> > If you decide to go that route it would probably be easiest to start back
> > over from a security perspective, meaning:
> >
> > - Stop all the nodes and delete the users.xml and authorizations.xml from
> > all nodes
> >
> > - Configure authorizers.xml with the appropriate initial admin (or legacy
> > file) and node identities based on the new certs
> >
> > - Ensure authorizers.xml is the same on all nodes
> >
> > - Then restart everything
> >
> >
> >
> > Alternatively, you might be able to manually add users for all of the new
> > certs before shutting everything down and give them the appropriate
> > policies, then restart everything, but requires more manual work to get
> > everything lined up.
> >
> >
> >
> >
> >
> > On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton
> > <co...@secdata.com> wrote:
> >
> > Hi,
> >
> > As a plan for tomorrow – I have generated new keystores, truststores, client
> > certts etc. for all nodes in my cluster using the
> >
> >
> >
> > From: Bryan Bende <bb...@gmail.com>
> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Date: Wednesday, 19 October 2016 at 15:33
> >
> >
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> >
> >
> > Trying to think of things to check here...
> >
> >
> >
> > Does every node have nifi.remote.input.secure=true in nifi.properties and
> > the URL in the RPG is an https URL?
> >
> >
> >
> > On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton
> > <co...@secdata.com> wrote:
> >
> > One other thing…
> >
> > The RPGs have an unlocked padlock on them saying S2S is not secure.
> >
> > Conrad
> >
> >
> >
> > From: Bryan Bende <bb...@gmail.com>
> > Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Date: Wednesday, 19 October 2016 at 15:20
> > To: "users@nifi.apache.org" <us...@nifi.apache.org>
> > Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> >
> >
> >
> > Ok that does seem like a TLS/SSL issue...
> >
> >
> >
> > Is this a single cluster doing site-to-site to itself?
> >
> >
> >
> > On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com> wrote:
> >
> > thanks conrad - did get it. Bryan is being more helpful that I so I
> > went silent :-)
> >
> > On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
> >
> > <co...@secdata.com> wrote:
> >> Hi Joe,
> >> Yep,
> >> Tried removing the RPG that referenced the NCM and adding new one with
> >> one of the nifis as url.
> >> That sort of worked, but kept getting errors about the NCM not being
> >> available for the ports and therefore couldn’t actually enable the port I
> >> needed to for that RPG.
> >> Thanks
> >> Conrad
> >>
> >> (sending again as don’t know if the stupid header ‘spoofed’ is stopping
> >> getting though – apologies if already sent)
> >>
> >> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
> >>
> >> Conrad,
> >>
> >> For s2s now you can just point at any of the nodes in the cluster.
> >> Have you tried changing the URL or removing and adding new RPG
> >> entries?
> >>
> >> Thanks
> >> Joe
> >>
> >> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> >> <co...@secdata.com> wrote:
> >> > Hi,
> >> >
> >> > I have finally taken the plunge to upgrade my cluster from 0.6.1
> >> to 1.0.0.
> >> >
> >> > 6 nodes with a NCM.
> >> >
> >> > With the removal of NCM in 1.0.0 I believe I now have an issue
> >> where none of
> >> > my Remote Process Groups work as they previously did because
> >> they were
> >> > configured to connect to the NCM (as the RPG url) which now
> >> doesn’t exist.
> >> >
> >> > I have tried converting my NCM to a node but whilst I can get it
> >> running
> >> > (sort of) when I try and connect to the cluster I get something
> >> like this in
> >> > my logs…
> >> >
> >> >
> >> >
> >> > 2016-10-19 13:14:44,109 ERROR [main]
> >> o.a.nifi.controller.StandardFlowService
> >> > Failed to load flow from cluster due to:
> >> > org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> connect
> >> > node to cluster because local flow is different than cluster
> >> flow.
> >> >
> >> > org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> connect
> >> > node to cluster because local flow is different than cluster
> >> flow.
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> >> > [nifi-jetty-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > Caused by:
> >> org.apache.nifi.controller.UninheritableFlowException: Proposed
> >> > Authorizer is not inheritable by the flow controller because of
> >> Authorizer
> >> > differences: Proposed Authorizations do not match current
> >> Authorizations
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > ... 4 common frames omitted
> >> >
> >> > 2016-10-19 13:14:44,414 ERROR [main]
> >> o.a.n.c.c.node.NodeClusterCoordinator
> >> > Event Reported for ncm-cm1.local:9090 -- Node disconnected from
> >> > cluster due to
> >> org.apache.nifi.controller.UninheritableFlowException: Failed
> >> > to connect node to cluster because local flow is different than
> >> cluster
> >> > flow.
> >> >
> >> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> >> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
> >> Thread[Shutdown
> >> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >> >
> >> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> >> > org.apache.nifi.NiFi
> >> >
> >> > java.lang.NullPointerException: null
> >> >
> >> > at
> >> >
> >> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> >> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >> >
> >> > 2016-10-19 13:14:44,448 WARN [main]
> >> o.a.n.c.l.e.CuratorLeaderElectionManager
> >> > Failed to close Leader Selector for Cluster Coordinator
> >> >
> >> > java.lang.IllegalStateException: Already closed or has not been
> >> started
> >> >
> >> > at
> >> >
> >> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> >> > ~[guava-18.0.jar:na]
> >> >
> >> > at
> >> >
> >> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> >> > ~[curator-recipes-2.11.0.jar:na]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> >> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> >> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> >> > [nifi-jetty-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> >> > org.apache.nifi.io.socket.SocketListener Failed to communicate
> >> with Unknown
> >> > Host due to java.net.SocketException: Socket closed
> >> >
> >> > java.net.SocketException: Socket closed
> >> >
> >> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> java.net.ServerSocket.implAccept(ServerSocket.java:545)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> >> > ~[na:1.8.0_51]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> >> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >> >
> >> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >> >
> >> > 2016-10-19 13:14:45,064 WARN [main]
> >> org.apache.nifi.web.server.JettyServer
> >> > Failed to start web server... shutting down.
> >> >
> >> > java.lang.Exception: Unable to load flow due to:
> >> java.io.IOException:
> >> > org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> connect
> >> > node to cluster because local flow is different than cluster
> >> flow.
> >> >
> >> > at
> >> >
> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> >> > [nifi-runtime-1.0.0.jar:1.0.0]
> >> >
> >> > Caused by: java.io.IOException:
> >> > org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> connect
> >> > node to cluster because local flow is different than cluster
> >> flow.
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> >> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >> >
> >> > ... 2 common frames omitted
> >> >
> >> > Caused by:
> >> org.apache.nifi.controller.UninheritableFlowException: Failed to
> >> > connect node to cluster because local flow is different than
> >> cluster flow.
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > ... 3 common frames omitted
> >> >
> >> > Caused by:
> >> org.apache.nifi.controller.UninheritableFlowException: Proposed
> >> > Authorizer is not inheritable by the flow controller because of
> >> Authorizer
> >> > differences: Proposed Authorizations do not match current
> >> Authorizations
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > at
> >> >
> >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> >> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >> >
> >> > ... 4 common frames omitted
> >> >
> >> > [root@ncm-cm1 logs]#
> >> >
> >> >
> >> >
> >> > I don’t know if the ‘Proposed Authorizer is not inheritable…’
> >> exception is
> >> > part of the problem too.
> >> >
> >> > The docs weren’t very clear on whether (when upgrading and using
> >> the legacy
> >> > support of the authorized-user.xml path required the nodes to be
> >> also added
> >> > to the authorizers.xml.
> >> >
> >> > I did add them in the end as various attempts to get the cluster
> >> up and
> >> > running without them failed (as each server didn’t seem to have
> >> rights to do
> >> > anything.
> >> >
> >> >
> >> >
> >> > I have a lot of RPG in my work flows as I am ingesting many
> >> syslog data
> >> > sources and this was the recommended pattern to distribute the
> >> data
> >> > (listensyslog…run on primary, output to port (RPG), pick up in
> >> rest of data
> >> > flow),
> >> >
> >> >
> >> >
> >> > Any suggestions on where to start trying to get this working?
> >> >
> >> > I’ve tried creating a new RPG on one on the nifis and connecting
> >> the
> >> > syslog to that which sort of worked but then I have a bunch of
> >> other errors
> >> > when trying to enable the ports to do with not being able to
> >> connect to
> >> > (what was) the NCM.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Conrad
> >> >
> >> >
> >> >
> >> > SecureData, combating cyber threats
> >> >
> >> > ________________________________
> >> >
> >> > The information contained in this message or any of its
> >> attachments may be
> >> > privileged and confidential and intended for the exclusive use
> >> of the
> >> > intended recipient. If you are not the intended recipient any
> >> disclosure,
> >> > reproduction, distribution or other dissemination or use of this
> >> > communications is strictly prohibited. The views expressed in
> >> this email are
> >> > those of the individual and not necessarily of SecureData Europe
> >> Ltd. Any
> >> > prices quoted are only valid if followed up by a formal written
> >> quote.
> >> >
> >> > SecureData Europe Limited. Registered in England & Wales
> >> 04365896.
> >> > Registered Address: SecureData House, Hermitage Court, Hermitage
> >> Lane,
> >> > Maidstone, Kent, ME16 9NT
> >>
> >>
> >> ***This email originated outside SecureData***
> >>
> >> Click
> >> https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==
> >> to report this email as spam.
> >>
> >>
> >>
> >>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > * Unknown Key
> > * 0x2F7DEF69
> >
> >
> >
> >
> >
> > * Unknown Key
> > * 0x2F7DEF69
> >
> >
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
I’ve just tried to replicate but without success. Using a RPG with RAW transfer protocol.
If you want I can try removing the additional lines in my nifi.properties (the ones for http.port and host that I added which appeared to get this working?
Regards
Conrad
On 21/10/2016, 15:24, "Joe Witt" <jo...@gmail.com> wrote:
Whoa there.... can you turn on debug logging in conf/logback.xml by adding
<logger name="org.apache.nifi.remote.SocketRemoteSiteListener "
level="DEBUG"/>
Can you submit a JIRA for the log output with the stacktrace for those
NPEs please.
Thanks
Joe
On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
<co...@secdata.com> wrote:
> Hi,
>
> I realise this may be getting boring for most but…
>
> I didn’t find any resolution to the upgrade so I have cleanly installed
> v1.0.0 and oddly experienced the same issue with RPGs in that although the
> RPGs didn’t show any errors (in so much as they had no warning on them and
> reported that S2S was secure) the errors reported were about not being able
> to determine other nodes in cluster.
>
> This is a section of log that showed an error that I don’t think I saw
> before but only including here in case one of you fine folks need it for any
> reason…
>
>
>
> ERROR [Site-to-Site Worker Thread-194]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode3-cm1.mis.local:57289]
> (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c8-bffc59d650a7])
> due to java.lang.NullPointerException; closing connection
>
> 2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode1-cm1.mis.local:35831]
> (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c9-d294bb4037f5])
> due to java.lang.NullPointerException; closing connection
>
> 2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode5-cm1.mis.local:59093]
> (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-8201-16b905e5beb6])
> due to java.lang.NullPointerException; closing connection
>
> 2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode6-cm1.mis.local:37275]
> (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a149-4892d0193819])
> due to java.lang.NullPointerException; closing connection
>
> 2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode4-cm1.mis.local:57745]
> (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-8047-b5ba4d9b67ec])
> due to java.lang.NullPointerException; closing connection
>
>
>
> in my naivety this suggests a problem with the socket (RAW) connection
> protocol. The relevant section for S2S connection in my nifi.properties is
>
> nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the number
> different for each node obviously
>
> nifi.remote.input.socket.port=10443
>
> nifi.remote.input.secure=true
>
> nifi.remote.input.http.enabled=false
>
> nifi.remote.input.http.transaction.ttl=30 sec
>
>
>
> the errors suggest that the port specified here aren’t being used, but some
> random ports are being used instead. Of course this is complete supposition
> and probably a red herring.
>
>
>
> Anyway, I updated my nifi.properties to
>
> nifi.remote.input.socket.host=nifinode6-cm1.mis.local
>
> nifi.remote.input.http.host=nifinode6-cm1.mis.local
>
> nifi.remote.input.socket.port=10443
>
> nifi.remote.input.http.port=11443
>
> nifi.remote.input.secure=true
>
> nifi.remote.input.http.enabled=true
>
> nifi.remote.input.http.transaction.ttl=30 sec
>
>
>
> and used HTTP for my RPG and is now working ok.
>
>
>
> The test harness for this is
>
>
>
> GenerateFlowFile->RGP(test input port)
>
> InputPort(test input)->LogAttribute
>
>
>
> Regards,
>
> Conrad
>
>
>
>
>
>
>
> From: Conrad Crampton <co...@SecData.com>
> Date: Friday, 21 October 2016 at 08:18
>
>
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Hi,
>
> Yes, the access policy for the input port at the root level of my workspace
> has S2S access policy (receive data via site to site) for all server nodes
> (I have all my nodes in one group).
>
> For the next level down in my workspace (I have other process groups from my
> main ‘root’ page in the UI space to organise the flows), I have input ports
> on the next level down of flows which when I check the access policies on
> that for S2S, the receive and send data via site-to-site options are greyed
> out and if I try to override the policy, they still are. I don’t know if
> this is important, but from reading the post below, the issue that the
> access policies looks to address is different from my issue. The RPG has a
> locked padlock and says site to site is secure. I don’t have any warning
> triangles on the RPG itself, but I have the aforementioned warnings in my
> logs.
>
>
>
> I’m going to abandon this now as I can’t get it to work.
>
>
>
> I’m going to try a fresh install and go with that – and have to recreate all
> my flows where necessary. I’m moving to a new model of data ingestion anyway
> so isn’t as catastrophic as it might be.
>
>
>
> Thanks for the help.
>
> Conrad
>
>
>
> From: Andy LoPresto <al...@apache.org>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Thursday, 20 October 2016 at 17:31
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> * PGP Signed by an unknown key
>
> Conrad,
>
>
>
> For the site-to-site did you follow the instructions here [1]? Each node
> needs to be added as a user in order to make the connections.
>
>
>
> [1]
> http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> alopresto.apache@gmail.com
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Oct 20, 2016, at 7:36 AM, Conrad Crampton <co...@SecData.com>
> wrote:
>
>
>
> Ok,
>
> So I tried everything suggested so far to no avail unfortunately.
>
>
>
> So what I have done is to create all new certs etc. using the tookit.
> Updated my existing authoriszed-users.xml to have to match the full cert
> distinguished names CN=server.name, OU=NIFI etc.
>
>
>
> Recreated all my remote process groups to not reference the original NCM as
> that still wouldn’t work – after a complete new install (upgrade).
>
>
>
> So now what I have is a six node cluster using original data/worker nodes
> and they are part of the cluster – all appears to be working ie. I can log
> into the UI (nice by the way ;-) on each server. There are no SSL handshake
> errors etc. BUT the RPG (newly created) still don’t appear to be working. I
> am getting
>
>
>
> 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>
> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>
> nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>
> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>
> nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>
> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>
> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>
> nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
>
>
> I can telnet from server to server on both https (UI) port and S2S port.
>
> I am really at a loss as to what to do now.
>
>
>
> Data is queuing up in my input processors with nowhere to go.
>
> Do I have to do something radical here to get this working like stopping
> everything, clearing out all the queues then starting up again??? I really
> don’t want to do this obviously but I am getting nowhere on this – two days
> of frustration with nothing to show for it.
>
>
>
> Any more suggestions please??
>
> Thanks for your patience.
>
> Conrad
>
>
>
>
>
> From: Andy LoPresto <al...@apache.org>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Wednesday, 19 October 2016 at 18:24
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
>> Old Signed by an unknown key
>
> Hi Conrad,
>
>
>
> Bryan is correct that changing the certificates (and the encapsulating
> keystores and truststores) will not affect any data held in the nodes.
>
>
>
> Regenerating everything using the TLS toolkit should hopefully not be too
> challenging, but I am also curious as to why you are getting these handshake
> exceptions now. As Bryan pointed out, adding the following line to
> bootstrap.conf will provide substantial additional log output which should
> help trace the issue.
>
>
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
>
>
> You can also imitate the node connecting to the (previous) NCM via this
> command:
>
>
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
>
>
> Where:
>
>
>
> <host:port> = the hostname and port of the “NCM”
> <path_to_your_cert.pem> = the public key used to identify the “node” (can be
> exported from the node keystore [1])
> <path_to_your_key.pem> = the private key used to identify the “node” (can be
> exported from the node keystore via 2 step process)
> <path_to_your_CA_cert.pem> = the public key used to sign the “NCM”
> certificate (could be a 3rd party like Verisign or DigiCert, or an internal
> organization CA if you have one)
>
>
>
> If you’ve already regenerated everything and it works, that’s fine. But if
> you have the time to try and investigate the old certs, we are interested
> and prepared to help. Thanks.
>
>
>
> [1] https://security.stackexchange.com/a/66865/16485
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> alopresto.apache@gmail.com
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com> wrote:
>
>
>
> That is definitely weird that it is only an issue on the node that used to
> be the NCM.
>
> Might be worth double checking the keystore and truststore of that one node
> and make sure it has what you would expect, and also double check
> nifi.properties compared to the others to see if anything seems different.
>
>
>
> Changing all of the keystores, truststores, etc should be fine from a data
> perspective...
>
>
>
> If you decide to go that route it would probably be easiest to start back
> over from a security perspective, meaning:
>
> - Stop all the nodes and delete the users.xml and authorizations.xml from
> all nodes
>
> - Configure authorizers.xml with the appropriate initial admin (or legacy
> file) and node identities based on the new certs
>
> - Ensure authorizers.xml is the same on all nodes
>
> - Then restart everything
>
>
>
> Alternatively, you might be able to manually add users for all of the new
> certs before shutting everything down and give them the appropriate
> policies, then restart everything, but requires more manual work to get
> everything lined up.
>
>
>
>
>
> On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton
> <co...@secdata.com> wrote:
>
> Hi,
>
> As a plan for tomorrow – I have generated new keystores, truststores, client
> certts etc. for all nodes in my cluster using the
>
>
>
> From: Bryan Bende <bb...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Wednesday, 19 October 2016 at 15:33
>
>
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
>
>
> Trying to think of things to check here...
>
>
>
> Does every node have nifi.remote.input.secure=true in nifi.properties and
> the URL in the RPG is an https URL?
>
>
>
> On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton
> <co...@secdata.com> wrote:
>
> One other thing…
>
> The RPGs have an unlocked padlock on them saying S2S is not secure.
>
> Conrad
>
>
>
> From: Bryan Bende <bb...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Wednesday, 19 October 2016 at 15:20
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Ok that does seem like a TLS/SSL issue...
>
>
>
> Is this a single cluster doing site-to-site to itself?
>
>
>
> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com> wrote:
>
> thanks conrad - did get it. Bryan is being more helpful that I so I
> went silent :-)
>
> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>
> <co...@secdata.com> wrote:
>> Hi Joe,
>> Yep,
>> Tried removing the RPG that referenced the NCM and adding new one with
>> one of the nifis as url.
>> That sort of worked, but kept getting errors about the NCM not being
>> available for the ports and therefore couldn’t actually enable the port I
>> needed to for that RPG.
>> Thanks
>> Conrad
>>
>> (sending again as don’t know if the stupid header ‘spoofed’ is stopping
>> getting though – apologies if already sent)
>>
>> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
>>
>> Conrad,
>>
>> For s2s now you can just point at any of the nodes in the cluster.
>> Have you tried changing the URL or removing and adding new RPG
>> entries?
>>
>> Thanks
>> Joe
>>
>> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
>> <co...@secdata.com> wrote:
>> > Hi,
>> >
>> > I have finally taken the plunge to upgrade my cluster from 0.6.1
>> to 1.0.0.
>> >
>> > 6 nodes with a NCM.
>> >
>> > With the removal of NCM in 1.0.0 I believe I now have an issue
>> where none of
>> > my Remote Process Groups work as they previously did because
>> they were
>> > configured to connect to the NCM (as the RPG url) which now
>> doesn’t exist.
>> >
>> > I have tried converting my NCM to a node but whilst I can get it
>> running
>> > (sort of) when I try and connect to the cluster I get something
>> like this in
>> > my logs…
>> >
>> >
>> >
>> > 2016-10-19 13:14:44,109 ERROR [main]
>> o.a.nifi.controller.StandardFlowService
>> > Failed to load flow from cluster due to:
>> > org.apache.nifi.controller.UninheritableFlowException: Failed to
>> connect
>> > node to cluster because local flow is different than cluster
>> flow.
>> >
>> > org.apache.nifi.controller.UninheritableFlowException: Failed to
>> connect
>> > node to cluster because local flow is different than cluster
>> flow.
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> > [nifi-jetty-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > Caused by:
>> org.apache.nifi.controller.UninheritableFlowException: Proposed
>> > Authorizer is not inheritable by the flow controller because of
>> Authorizer
>> > differences: Proposed Authorizations do not match current
>> Authorizations
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > ... 4 common frames omitted
>> >
>> > 2016-10-19 13:14:44,414 ERROR [main]
>> o.a.n.c.c.node.NodeClusterCoordinator
>> > Event Reported for ncm-cm1.local:9090 -- Node disconnected from
>> > cluster due to
>> org.apache.nifi.controller.UninheritableFlowException: Failed
>> > to connect node to cluster because local flow is different than
>> cluster
>> > flow.
>> >
>> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
>> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
>> Thread[Shutdown
>> > Cluster Coordinator,5,main]: java.lang.NullPointerException
>> >
>> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
>> > org.apache.nifi.NiFi
>> >
>> > java.lang.NullPointerException: null
>> >
>> > at
>> >
>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
>> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
>> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
>> >
>> > 2016-10-19 13:14:44,448 WARN [main]
>> o.a.n.c.l.e.CuratorLeaderElectionManager
>> > Failed to close Leader Selector for Cluster Coordinator
>> >
>> > java.lang.IllegalStateException: Already closed or has not been
>> started
>> >
>> > at
>> >
>> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
>> > ~[guava-18.0.jar:na]
>> >
>> > at
>> >
>> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
>> > ~[curator-recipes-2.11.0.jar:na]
>> >
>> > at
>> >
>> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
>> > [nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
>> > [nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
>> > [nifi-jetty-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
>> > org.apache.nifi.io.socket.SocketListener Failed to communicate
>> with Unknown
>> > Host due to java.net.SocketException: Socket closed
>> >
>> > java.net.SocketException: Socket closed
>> >
>> > at java.net.PlainSocketImpl.socketAccept(Native Method)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
>> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>> >
>> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
>> >
>> > 2016-10-19 13:14:45,064 WARN [main]
>> org.apache.nifi.web.server.JettyServer
>> > Failed to start web server... shutting down.
>> >
>> > java.lang.Exception: Unable to load flow due to:
>> java.io.IOException:
>> > org.apache.nifi.controller.UninheritableFlowException: Failed to
>> connect
>> > node to cluster because local flow is different than cluster
>> flow.
>> >
>> > at
>> >
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
>> > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > Caused by: java.io.IOException:
>> > org.apache.nifi.controller.UninheritableFlowException: Failed to
>> connect
>> > node to cluster because local flow is different than cluster
>> flow.
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> >
>> > ... 2 common frames omitted
>> >
>> > Caused by:
>> org.apache.nifi.controller.UninheritableFlowException: Failed to
>> > connect node to cluster because local flow is different than
>> cluster flow.
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > ... 3 common frames omitted
>> >
>> > Caused by:
>> org.apache.nifi.controller.UninheritableFlowException: Proposed
>> > Authorizer is not inheritable by the flow controller because of
>> Authorizer
>> > differences: Proposed Authorizations do not match current
>> Authorizations
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > ... 4 common frames omitted
>> >
>> > [root@ncm-cm1 logs]#
>> >
>> >
>> >
>> > I don’t know if the ‘Proposed Authorizer is not inheritable…’
>> exception is
>> > part of the problem too.
>> >
>> > The docs weren’t very clear on whether (when upgrading and using
>> the legacy
>> > support of the authorized-user.xml path required the nodes to be
>> also added
>> > to the authorizers.xml.
>> >
>> > I did add them in the end as various attempts to get the cluster
>> up and
>> > running without them failed (as each server didn’t seem to have
>> rights to do
>> > anything.
>> >
>> >
>> >
>> > I have a lot of RPG in my work flows as I am ingesting many
>> syslog data
>> > sources and this was the recommended pattern to distribute the
>> data
>> > (listensyslog…run on primary, output to port (RPG), pick up in
>> rest of data
>> > flow),
>> >
>> >
>> >
>> > Any suggestions on where to start trying to get this working?
>> >
>> > I’ve tried creating a new RPG on one on the nifis and connecting
>> the
>> > syslog to that which sort of worked but then I have a bunch of
>> other errors
>> > when trying to enable the ports to do with not being able to
>> connect to
>> > (what was) the NCM.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Conrad
>> >
>> >
>> >
>> > SecureData, combating cyber threats
>> >
>> > ________________________________
>> >
>> > The information contained in this message or any of its
>> attachments may be
>> > privileged and confidential and intended for the exclusive use
>> of the
>> > intended recipient. If you are not the intended recipient any
>> disclosure,
>> > reproduction, distribution or other dissemination or use of this
>> > communications is strictly prohibited. The views expressed in
>> this email are
>> > those of the individual and not necessarily of SecureData Europe
>> Ltd. Any
>> > prices quoted are only valid if followed up by a formal written
>> quote.
>> >
>> > SecureData Europe Limited. Registered in England & Wales
>> 04365896.
>> > Registered Address: SecureData House, Hermitage Court, Hermitage
>> Lane,
>> > Maidstone, Kent, ME16 9NT
>>
>>
>> ***This email originated outside SecureData***
>>
>> Click
>> https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==
>> to report this email as spam.
>>
>>
>>
>>
>
>
>
>
>
>
>
>
>
> * Unknown Key
> * 0x2F7DEF69
>
>
>
>
>
> * Unknown Key
> * 0x2F7DEF69
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by ma...@hotmail.com.
Joe,
I'm not at a computer right now so can't easily find the JIRA, but I think there already is one for this. If you enable raw site-to-site without http (or was it http without raw site-to-site?) then it throws a NPE. This should be fixed already on master, if I am not mistaken.
-Mark
Sent from my iPhone
> On Oct 21, 2016, at 9:27 AM, Joe Witt <jo...@gmail.com> wrote:
>
> Whoa there.... can you turn on debug logging in conf/logback.xml by adding
>
> <logger name="org.apache.nifi.remote.SocketRemoteSiteListener "
> level="DEBUG"/>
>
> Can you submit a JIRA for the log output with the stacktrace for those
> NPEs please.
>
> Thanks
> Joe
>
> On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
> <co...@secdata.com> wrote:
>> Hi,
>>
>> I realise this may be getting boring for most but…
>>
>> I didn’t find any resolution to the upgrade so I have cleanly installed
>> v1.0.0 and oddly experienced the same issue with RPGs in that although the
>> RPGs didn’t show any errors (in so much as they had no warning on them and
>> reported that S2S was secure) the errors reported were about not being able
>> to determine other nodes in cluster.
>>
>> This is a section of log that showed an error that I don’t think I saw
>> before but only including here in case one of you fine folks need it for any
>> reason…
>>
>>
>>
>> ERROR [Site-to-Site Worker Thread-194]
>> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
>> instance Peer[url=nifi://nifinode3-cm1.mis.local:57289]
>> (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c8-bffc59d650a7])
>> due to java.lang.NullPointerException; closing connection
>>
>> 2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195]
>> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
>> instance Peer[url=nifi://nifinode1-cm1.mis.local:35831]
>> (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c9-d294bb4037f5])
>> due to java.lang.NullPointerException; closing connection
>>
>> 2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196]
>> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
>> instance Peer[url=nifi://nifinode5-cm1.mis.local:59093]
>> (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-8201-16b905e5beb6])
>> due to java.lang.NullPointerException; closing connection
>>
>> 2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197]
>> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
>> instance Peer[url=nifi://nifinode6-cm1.mis.local:37275]
>> (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a149-4892d0193819])
>> due to java.lang.NullPointerException; closing connection
>>
>> 2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198]
>> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
>> instance Peer[url=nifi://nifinode4-cm1.mis.local:57745]
>> (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-8047-b5ba4d9b67ec])
>> due to java.lang.NullPointerException; closing connection
>>
>>
>>
>> in my naivety this suggests a problem with the socket (RAW) connection
>> protocol. The relevant section for S2S connection in my nifi.properties is
>>
>> nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the number
>> different for each node obviously
>>
>> nifi.remote.input.socket.port=10443
>>
>> nifi.remote.input.secure=true
>>
>> nifi.remote.input.http.enabled=false
>>
>> nifi.remote.input.http.transaction.ttl=30 sec
>>
>>
>>
>> the errors suggest that the port specified here aren’t being used, but some
>> random ports are being used instead. Of course this is complete supposition
>> and probably a red herring.
>>
>>
>>
>> Anyway, I updated my nifi.properties to
>>
>> nifi.remote.input.socket.host=nifinode6-cm1.mis.local
>>
>> nifi.remote.input.http.host=nifinode6-cm1.mis.local
>>
>> nifi.remote.input.socket.port=10443
>>
>> nifi.remote.input.http.port=11443
>>
>> nifi.remote.input.secure=true
>>
>> nifi.remote.input.http.enabled=true
>>
>> nifi.remote.input.http.transaction.ttl=30 sec
>>
>>
>>
>> and used HTTP for my RPG and is now working ok.
>>
>>
>>
>> The test harness for this is
>>
>>
>>
>> GenerateFlowFile->RGP(test input port)
>>
>> InputPort(test input)->LogAttribute
>>
>>
>>
>> Regards,
>>
>> Conrad
>>
>>
>>
>>
>>
>>
>>
>> From: Conrad Crampton <co...@SecData.com>
>> Date: Friday, 21 October 2016 at 08:18
>>
>>
>> To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>>
>>
>>
>> Hi,
>>
>> Yes, the access policy for the input port at the root level of my workspace
>> has S2S access policy (receive data via site to site) for all server nodes
>> (I have all my nodes in one group).
>>
>> For the next level down in my workspace (I have other process groups from my
>> main ‘root’ page in the UI space to organise the flows), I have input ports
>> on the next level down of flows which when I check the access policies on
>> that for S2S, the receive and send data via site-to-site options are greyed
>> out and if I try to override the policy, they still are. I don’t know if
>> this is important, but from reading the post below, the issue that the
>> access policies looks to address is different from my issue. The RPG has a
>> locked padlock and says site to site is secure. I don’t have any warning
>> triangles on the RPG itself, but I have the aforementioned warnings in my
>> logs.
>>
>>
>>
>> I’m going to abandon this now as I can’t get it to work.
>>
>>
>>
>> I’m going to try a fresh install and go with that – and have to recreate all
>> my flows where necessary. I’m moving to a new model of data ingestion anyway
>> so isn’t as catastrophic as it might be.
>>
>>
>>
>> Thanks for the help.
>>
>> Conrad
>>
>>
>>
>> From: Andy LoPresto <al...@apache.org>
>> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Date: Thursday, 20 October 2016 at 17:31
>> To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>>
>>
>>
>> * PGP Signed by an unknown key
>>
>> Conrad,
>>
>>
>>
>> For the site-to-site did you follow the instructions here [1]? Each node
>> needs to be added as a user in order to make the connections.
>>
>>
>>
>> [1]
>> http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>>
>>
>>
>> Andy LoPresto
>>
>> alopresto@apache.org
>>
>> alopresto.apache@gmail.com
>>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>>
>>
>> On Oct 20, 2016, at 7:36 AM, Conrad Crampton <co...@SecData.com>
>> wrote:
>>
>>
>>
>> Ok,
>>
>> So I tried everything suggested so far to no avail unfortunately.
>>
>>
>>
>> So what I have done is to create all new certs etc. using the tookit.
>> Updated my existing authoriszed-users.xml to have to match the full cert
>> distinguished names CN=server.name, OU=NIFI etc.
>>
>>
>>
>> Recreated all my remote process groups to not reference the original NCM as
>> that still wouldn’t work – after a complete new install (upgrade).
>>
>>
>>
>> So now what I have is a six node cluster using original data/worker nodes
>> and they are part of the cluster – all appears to be working ie. I can log
>> into the UI (nice by the way ;-) on each server. There are no SSL handshake
>> errors etc. BUT the RPG (newly created) still don’t appear to be working. I
>> am getting
>>
>>
>>
>> 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>>
>> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623
>> Unable to refresh Remote Group's peers due to Unable to communicate with
>> remote NiFi cluster in order to determine which nodes exist in the remote
>> cluster
>>
>> 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>>
>> nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274
>> Unable to refresh Remote Group's peers due to Unable to communicate with
>> remote NiFi cluster in order to determine which nodes exist in the remote
>> cluster
>>
>> 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>>
>> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1
>> Unable to refresh Remote Group's peers due to Unable to communicate with
>> remote NiFi cluster in order to determine which nodes exist in the remote
>> cluster
>>
>> 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>>
>> nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c
>> Unable to refresh Remote Group's peers due to Unable to communicate with
>> remote NiFi cluster in order to determine which nodes exist in the remote
>> cluster
>>
>> 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>>
>> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef
>> Unable to refresh Remote Group's peers due to Unable to communicate with
>> remote NiFi cluster in order to determine which nodes exist in the remote
>> cluster
>>
>> 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>>
>> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037
>> Unable to refresh Remote Group's peers due to Unable to communicate with
>> remote NiFi cluster in order to determine which nodes exist in the remote
>> cluster
>>
>> 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>>
>> nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426
>> Unable to refresh Remote Group's peers due to Unable to communicate with
>> remote NiFi cluster in order to determine which nodes exist in the remote
>> cluster
>>
>>
>>
>> I can telnet from server to server on both https (UI) port and S2S port.
>>
>> I am really at a loss as to what to do now.
>>
>>
>>
>> Data is queuing up in my input processors with nowhere to go.
>>
>> Do I have to do something radical here to get this working like stopping
>> everything, clearing out all the queues then starting up again??? I really
>> don’t want to do this obviously but I am getting nowhere on this – two days
>> of frustration with nothing to show for it.
>>
>>
>>
>> Any more suggestions please??
>>
>> Thanks for your patience.
>>
>> Conrad
>>
>>
>>
>>
>>
>> From: Andy LoPresto <al...@apache.org>
>> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Date: Wednesday, 19 October 2016 at 18:24
>> To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>>
>>
>>
>>> Old Signed by an unknown key
>>
>> Hi Conrad,
>>
>>
>>
>> Bryan is correct that changing the certificates (and the encapsulating
>> keystores and truststores) will not affect any data held in the nodes.
>>
>>
>>
>> Regenerating everything using the TLS toolkit should hopefully not be too
>> challenging, but I am also curious as to why you are getting these handshake
>> exceptions now. As Bryan pointed out, adding the following line to
>> bootstrap.conf will provide substantial additional log output which should
>> help trace the issue.
>>
>>
>>
>> java.arg.15=-Djavax.net.debug=ssl,handshake
>>
>>
>>
>> You can also imitate the node connecting to the (previous) NCM via this
>> command:
>>
>>
>>
>> $ openssl s_client -connect <host:port> -debug -state -cert
>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>> <path_to_your_CA_cert.pem>
>>
>>
>>
>> Where:
>>
>>
>>
>> <host:port> = the hostname and port of the “NCM”
>> <path_to_your_cert.pem> = the public key used to identify the “node” (can be
>> exported from the node keystore [1])
>> <path_to_your_key.pem> = the private key used to identify the “node” (can be
>> exported from the node keystore via 2 step process)
>> <path_to_your_CA_cert.pem> = the public key used to sign the “NCM”
>> certificate (could be a 3rd party like Verisign or DigiCert, or an internal
>> organization CA if you have one)
>>
>>
>>
>> If you’ve already regenerated everything and it works, that’s fine. But if
>> you have the time to try and investigate the old certs, we are interested
>> and prepared to help. Thanks.
>>
>>
>>
>> [1] https://security.stackexchange.com/a/66865/16485
>>
>>
>>
>> Andy LoPresto
>>
>> alopresto@apache.org
>>
>> alopresto.apache@gmail.com
>>
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>>
>>
>> On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>>
>>
>> That is definitely weird that it is only an issue on the node that used to
>> be the NCM.
>>
>> Might be worth double checking the keystore and truststore of that one node
>> and make sure it has what you would expect, and also double check
>> nifi.properties compared to the others to see if anything seems different.
>>
>>
>>
>> Changing all of the keystores, truststores, etc should be fine from a data
>> perspective...
>>
>>
>>
>> If you decide to go that route it would probably be easiest to start back
>> over from a security perspective, meaning:
>>
>> - Stop all the nodes and delete the users.xml and authorizations.xml from
>> all nodes
>>
>> - Configure authorizers.xml with the appropriate initial admin (or legacy
>> file) and node identities based on the new certs
>>
>> - Ensure authorizers.xml is the same on all nodes
>>
>> - Then restart everything
>>
>>
>>
>> Alternatively, you might be able to manually add users for all of the new
>> certs before shutting everything down and give them the appropriate
>> policies, then restart everything, but requires more manual work to get
>> everything lined up.
>>
>>
>>
>>
>>
>> On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton
>> <co...@secdata.com> wrote:
>>
>> Hi,
>>
>> As a plan for tomorrow – I have generated new keystores, truststores, client
>> certts etc. for all nodes in my cluster using the
>>
>>
>>
>> From: Bryan Bende <bb...@gmail.com>
>> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Date: Wednesday, 19 October 2016 at 15:33
>>
>>
>> To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>>
>>
>>
>>
>>
>> Trying to think of things to check here...
>>
>>
>>
>> Does every node have nifi.remote.input.secure=true in nifi.properties and
>> the URL in the RPG is an https URL?
>>
>>
>>
>> On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton
>> <co...@secdata.com> wrote:
>>
>> One other thing…
>>
>> The RPGs have an unlocked padlock on them saying S2S is not secure.
>>
>> Conrad
>>
>>
>>
>> From: Bryan Bende <bb...@gmail.com>
>> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Date: Wednesday, 19 October 2016 at 15:20
>> To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>>
>>
>>
>> Ok that does seem like a TLS/SSL issue...
>>
>>
>>
>> Is this a single cluster doing site-to-site to itself?
>>
>>
>>
>> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>> thanks conrad - did get it. Bryan is being more helpful that I so I
>> went silent :-)
>>
>> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>>
>> <co...@secdata.com> wrote:
>>> Hi Joe,
>>> Yep,
>>> Tried removing the RPG that referenced the NCM and adding new one with
>>> one of the nifis as url.
>>> That sort of worked, but kept getting errors about the NCM not being
>>> available for the ports and therefore couldn’t actually enable the port I
>>> needed to for that RPG.
>>> Thanks
>>> Conrad
>>>
>>> (sending again as don’t know if the stupid header ‘spoofed’ is stopping
>>> getting though – apologies if already sent)
>>>
>>> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
>>>
>>> Conrad,
>>>
>>> For s2s now you can just point at any of the nodes in the cluster.
>>> Have you tried changing the URL or removing and adding new RPG
>>> entries?
>>>
>>> Thanks
>>> Joe
>>>
>>> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
>>> <co...@secdata.com> wrote:
>>>> Hi,
>>>>
>>>> I have finally taken the plunge to upgrade my cluster from 0.6.1
>>> to 1.0.0.
>>>>
>>>> 6 nodes with a NCM.
>>>>
>>>> With the removal of NCM in 1.0.0 I believe I now have an issue
>>> where none of
>>>> my Remote Process Groups work as they previously did because
>>> they were
>>>> configured to connect to the NCM (as the RPG url) which now
>>> doesn’t exist.
>>>>
>>>> I have tried converting my NCM to a node but whilst I can get it
>>> running
>>>> (sort of) when I try and connect to the cluster I get something
>>> like this in
>>>> my logs…
>>>>
>>>>
>>>>
>>>> 2016-10-19 13:14:44,109 ERROR [main]
>>> o.a.nifi.controller.StandardFlowService
>>>> Failed to load flow from cluster due to:
>>>> org.apache.nifi.controller.UninheritableFlowException: Failed to
>>> connect
>>>> node to cluster because local flow is different than cluster
>>> flow.
>>>>
>>>> org.apache.nifi.controller.UninheritableFlowException: Failed to
>>> connect
>>>> node to cluster because local flow is different than cluster
>>> flow.
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>>>> [nifi-jetty-1.0.0.jar:1.0.0]
>>>>
>>>> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>>>> [nifi-runtime-1.0.0.jar:1.0.0]
>>>>
>>>> at org.apache.nifi.NiFi.main(NiFi.java:243)
>>>> [nifi-runtime-1.0.0.jar:1.0.0]
>>>>
>>>> Caused by:
>>> org.apache.nifi.controller.UninheritableFlowException: Proposed
>>>> Authorizer is not inheritable by the flow controller because of
>>> Authorizer
>>>> differences: Proposed Authorizations do not match current
>>> Authorizations
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> ... 4 common frames omitted
>>>>
>>>> 2016-10-19 13:14:44,414 ERROR [main]
>>> o.a.n.c.c.node.NodeClusterCoordinator
>>>> Event Reported for ncm-cm1.local:9090 -- Node disconnected from
>>>> cluster due to
>>> org.apache.nifi.controller.UninheritableFlowException: Failed
>>>> to connect node to cluster because local flow is different than
>>> cluster
>>>> flow.
>>>>
>>>> 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
>>>> org.apache.nifi.NiFi An Unknown Error Occurred in Thread
>>> Thread[Shutdown
>>>> Cluster Coordinator,5,main]: java.lang.NullPointerException
>>>>
>>>> 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
>>>> org.apache.nifi.NiFi
>>>>
>>>> java.lang.NullPointerException: null
>>>>
>>>> at
>>>>
>>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
>>>> ~[na:1.8.0_51]
>>>>
>>>> at
>>>>
>>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
>>>> ~[na:1.8.0_51]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
>>>> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
>>>> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
>>>>
>>>> 2016-10-19 13:14:44,448 WARN [main]
>>> o.a.n.c.l.e.CuratorLeaderElectionManager
>>>> Failed to close Leader Selector for Cluster Coordinator
>>>>
>>>> java.lang.IllegalStateException: Already closed or has not been
>>> started
>>>>
>>>> at
>>>>
>>> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
>>>> ~[guava-18.0.jar:na]
>>>>
>>>> at
>>>>
>>> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
>>>> ~[curator-recipes-2.11.0.jar:na]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
>>>> [nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
>>>> [nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
>>>> [nifi-jetty-1.0.0.jar:1.0.0]
>>>>
>>>> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>>>> [nifi-runtime-1.0.0.jar:1.0.0]
>>>>
>>>> at org.apache.nifi.NiFi.main(NiFi.java:243)
>>>> [nifi-runtime-1.0.0.jar:1.0.0]
>>>>
>>>> 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
>>>> org.apache.nifi.io.socket.SocketListener Failed to communicate
>>> with Unknown
>>>> Host due to java.net.SocketException: Socket closed
>>>>
>>>> java.net.SocketException: Socket closed
>>>>
>>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>> ~[na:1.8.0_51]
>>>>
>>>> at
>>>>
>>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>>>> ~[na:1.8.0_51]
>>>>
>>>> at
>>> java.net.ServerSocket.implAccept(ServerSocket.java:545)
>>>> ~[na:1.8.0_51]
>>>>
>>>> at
>>>>
>>> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
>>>> ~[na:1.8.0_51]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
>>>> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>>>>
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
>>>>
>>>> 2016-10-19 13:14:45,064 WARN [main]
>>> org.apache.nifi.web.server.JettyServer
>>>> Failed to start web server... shutting down.
>>>>
>>>> java.lang.Exception: Unable to load flow due to:
>>> java.io.IOException:
>>>> org.apache.nifi.controller.UninheritableFlowException: Failed to
>>> connect
>>>> node to cluster because local flow is different than cluster
>>> flow.
>>>>
>>>> at
>>>>
>>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
>>>> ~[nifi-jetty-1.0.0.jar:1.0.0]
>>>>
>>>> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>>>> [nifi-runtime-1.0.0.jar:1.0.0]
>>>>
>>>> at org.apache.nifi.NiFi.main(NiFi.java:243)
>>>> [nifi-runtime-1.0.0.jar:1.0.0]
>>>>
>>>> Caused by: java.io.IOException:
>>>> org.apache.nifi.controller.UninheritableFlowException: Failed to
>>> connect
>>>> node to cluster because local flow is different than cluster
>>> flow.
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>>>> ~[nifi-jetty-1.0.0.jar:1.0.0]
>>>>
>>>> ... 2 common frames omitted
>>>>
>>>> Caused by:
>>> org.apache.nifi.controller.UninheritableFlowException: Failed to
>>>> connect node to cluster because local flow is different than
>>> cluster flow.
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> ... 3 common frames omitted
>>>>
>>>> Caused by:
>>> org.apache.nifi.controller.UninheritableFlowException: Proposed
>>>> Authorizer is not inheritable by the flow controller because of
>>> Authorizer
>>>> differences: Proposed Authorizations do not match current
>>> Authorizations
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> at
>>>>
>>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>>>> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>>>>
>>>> ... 4 common frames omitted
>>>>
>>>> [root@ncm-cm1 logs]#
>>>>
>>>>
>>>>
>>>> I don’t know if the ‘Proposed Authorizer is not inheritable…’
>>> exception is
>>>> part of the problem too.
>>>>
>>>> The docs weren’t very clear on whether (when upgrading and using
>>> the legacy
>>>> support of the authorized-user.xml path required the nodes to be
>>> also added
>>>> to the authorizers.xml.
>>>>
>>>> I did add them in the end as various attempts to get the cluster
>>> up and
>>>> running without them failed (as each server didn’t seem to have
>>> rights to do
>>>> anything.
>>>>
>>>>
>>>>
>>>> I have a lot of RPG in my work flows as I am ingesting many
>>> syslog data
>>>> sources and this was the recommended pattern to distribute the
>>> data
>>>> (listensyslog…run on primary, output to port (RPG), pick up in
>>> rest of data
>>>> flow),
>>>>
>>>>
>>>>
>>>> Any suggestions on where to start trying to get this working?
>>>>
>>>> I’ve tried creating a new RPG on one on the nifis and connecting
>>> the
>>>> syslog to that which sort of worked but then I have a bunch of
>>> other errors
>>>> when trying to enable the ports to do with not being able to
>>> connect to
>>>> (what was) the NCM.
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Conrad
>>>>
>>>>
>>>>
>>>> SecureData, combating cyber threats
>>>>
>>>> ________________________________
>>>>
>>>> The information contained in this message or any of its
>>> attachments may be
>>>> privileged and confidential and intended for the exclusive use
>>> of the
>>>> intended recipient. If you are not the intended recipient any
>>> disclosure,
>>>> reproduction, distribution or other dissemination or use of this
>>>> communications is strictly prohibited. The views expressed in
>>> this email are
>>>> those of the individual and not necessarily of SecureData Europe
>>> Ltd. Any
>>>> prices quoted are only valid if followed up by a formal written
>>> quote.
>>>>
>>>> SecureData Europe Limited. Registered in England & Wales
>>> 04365896.
>>>> Registered Address: SecureData House, Hermitage Court, Hermitage
>>> Lane,
>>>> Maidstone, Kent, ME16 9NT
>>>
>>>
>>> ***This email originated outside SecureData***
>>>
>>> Click
>>> https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==
>>> to report this email as spam.
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> * Unknown Key
>> * 0x2F7DEF69
>>
>>
>>
>>
>>
>> * Unknown Key
>> * 0x2F7DEF69
>>
>>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Joe Witt <jo...@gmail.com>.
Whoa there.... can you turn on debug logging in conf/logback.xml by adding
<logger name="org.apache.nifi.remote.SocketRemoteSiteListener "
level="DEBUG"/>
Can you submit a JIRA for the log output with the stacktrace for those
NPEs please.
Thanks
Joe
On Fri, Oct 21, 2016 at 10:21 AM, Conrad Crampton
<co...@secdata.com> wrote:
> Hi,
>
> I realise this may be getting boring for most but…
>
> I didn’t find any resolution to the upgrade so I have cleanly installed
> v1.0.0 and oddly experienced the same issue with RPGs in that although the
> RPGs didn’t show any errors (in so much as they had no warning on them and
> reported that S2S was secure) the errors reported were about not being able
> to determine other nodes in cluster.
>
> This is a section of log that showed an error that I don’t think I saw
> before but only including here in case one of you fine folks need it for any
> reason…
>
>
>
> ERROR [Site-to-Site Worker Thread-194]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode3-cm1.mis.local:57289]
> (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c8-bffc59d650a7])
> due to java.lang.NullPointerException; closing connection
>
> 2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode1-cm1.mis.local:35831]
> (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c9-d294bb4037f5])
> due to java.lang.NullPointerException; closing connection
>
> 2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode5-cm1.mis.local:59093]
> (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-8201-16b905e5beb6])
> due to java.lang.NullPointerException; closing connection
>
> 2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode6-cm1.mis.local:37275]
> (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a149-4892d0193819])
> due to java.lang.NullPointerException; closing connection
>
> 2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198]
> o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote
> instance Peer[url=nifi://nifinode4-cm1.mis.local:57745]
> (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-8047-b5ba4d9b67ec])
> due to java.lang.NullPointerException; closing connection
>
>
>
> in my naivety this suggests a problem with the socket (RAW) connection
> protocol. The relevant section for S2S connection in my nifi.properties is
>
> nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the number
> different for each node obviously
>
> nifi.remote.input.socket.port=10443
>
> nifi.remote.input.secure=true
>
> nifi.remote.input.http.enabled=false
>
> nifi.remote.input.http.transaction.ttl=30 sec
>
>
>
> the errors suggest that the port specified here aren’t being used, but some
> random ports are being used instead. Of course this is complete supposition
> and probably a red herring.
>
>
>
> Anyway, I updated my nifi.properties to
>
> nifi.remote.input.socket.host=nifinode6-cm1.mis.local
>
> nifi.remote.input.http.host=nifinode6-cm1.mis.local
>
> nifi.remote.input.socket.port=10443
>
> nifi.remote.input.http.port=11443
>
> nifi.remote.input.secure=true
>
> nifi.remote.input.http.enabled=true
>
> nifi.remote.input.http.transaction.ttl=30 sec
>
>
>
> and used HTTP for my RPG and is now working ok.
>
>
>
> The test harness for this is
>
>
>
> GenerateFlowFile->RGP(test input port)
>
> InputPort(test input)->LogAttribute
>
>
>
> Regards,
>
> Conrad
>
>
>
>
>
>
>
> From: Conrad Crampton <co...@SecData.com>
> Date: Friday, 21 October 2016 at 08:18
>
>
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Hi,
>
> Yes, the access policy for the input port at the root level of my workspace
> has S2S access policy (receive data via site to site) for all server nodes
> (I have all my nodes in one group).
>
> For the next level down in my workspace (I have other process groups from my
> main ‘root’ page in the UI space to organise the flows), I have input ports
> on the next level down of flows which when I check the access policies on
> that for S2S, the receive and send data via site-to-site options are greyed
> out and if I try to override the policy, they still are. I don’t know if
> this is important, but from reading the post below, the issue that the
> access policies looks to address is different from my issue. The RPG has a
> locked padlock and says site to site is secure. I don’t have any warning
> triangles on the RPG itself, but I have the aforementioned warnings in my
> logs.
>
>
>
> I’m going to abandon this now as I can’t get it to work.
>
>
>
> I’m going to try a fresh install and go with that – and have to recreate all
> my flows where necessary. I’m moving to a new model of data ingestion anyway
> so isn’t as catastrophic as it might be.
>
>
>
> Thanks for the help.
>
> Conrad
>
>
>
> From: Andy LoPresto <al...@apache.org>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Thursday, 20 October 2016 at 17:31
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> * PGP Signed by an unknown key
>
> Conrad,
>
>
>
> For the site-to-site did you follow the instructions here [1]? Each node
> needs to be added as a user in order to make the connections.
>
>
>
> [1]
> http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> alopresto.apache@gmail.com
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Oct 20, 2016, at 7:36 AM, Conrad Crampton <co...@SecData.com>
> wrote:
>
>
>
> Ok,
>
> So I tried everything suggested so far to no avail unfortunately.
>
>
>
> So what I have done is to create all new certs etc. using the tookit.
> Updated my existing authoriszed-users.xml to have to match the full cert
> distinguished names CN=server.name, OU=NIFI etc.
>
>
>
> Recreated all my remote process groups to not reference the original NCM as
> that still wouldn’t work – after a complete new install (upgrade).
>
>
>
> So now what I have is a six node cluster using original data/worker nodes
> and they are part of the cluster – all appears to be working ie. I can log
> into the UI (nice by the way ;-) on each server. There are no SSL handshake
> errors etc. BUT the RPG (newly created) still don’t appear to be working. I
> am getting
>
>
>
> 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>
> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>
> nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>
> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>
> nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
>
> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>
> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
> 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
>
> nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426
> Unable to refresh Remote Group's peers due to Unable to communicate with
> remote NiFi cluster in order to determine which nodes exist in the remote
> cluster
>
>
>
> I can telnet from server to server on both https (UI) port and S2S port.
>
> I am really at a loss as to what to do now.
>
>
>
> Data is queuing up in my input processors with nowhere to go.
>
> Do I have to do something radical here to get this working like stopping
> everything, clearing out all the queues then starting up again??? I really
> don’t want to do this obviously but I am getting nowhere on this – two days
> of frustration with nothing to show for it.
>
>
>
> Any more suggestions please??
>
> Thanks for your patience.
>
> Conrad
>
>
>
>
>
> From: Andy LoPresto <al...@apache.org>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Wednesday, 19 October 2016 at 18:24
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
>> Old Signed by an unknown key
>
> Hi Conrad,
>
>
>
> Bryan is correct that changing the certificates (and the encapsulating
> keystores and truststores) will not affect any data held in the nodes.
>
>
>
> Regenerating everything using the TLS toolkit should hopefully not be too
> challenging, but I am also curious as to why you are getting these handshake
> exceptions now. As Bryan pointed out, adding the following line to
> bootstrap.conf will provide substantial additional log output which should
> help trace the issue.
>
>
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
>
>
> You can also imitate the node connecting to the (previous) NCM via this
> command:
>
>
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
>
>
> Where:
>
>
>
> <host:port> = the hostname and port of the “NCM”
> <path_to_your_cert.pem> = the public key used to identify the “node” (can be
> exported from the node keystore [1])
> <path_to_your_key.pem> = the private key used to identify the “node” (can be
> exported from the node keystore via 2 step process)
> <path_to_your_CA_cert.pem> = the public key used to sign the “NCM”
> certificate (could be a 3rd party like Verisign or DigiCert, or an internal
> organization CA if you have one)
>
>
>
> If you’ve already regenerated everything and it works, that’s fine. But if
> you have the time to try and investigate the old certs, we are interested
> and prepared to help. Thanks.
>
>
>
> [1] https://security.stackexchange.com/a/66865/16485
>
>
>
> Andy LoPresto
>
> alopresto@apache.org
>
> alopresto.apache@gmail.com
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com> wrote:
>
>
>
> That is definitely weird that it is only an issue on the node that used to
> be the NCM.
>
> Might be worth double checking the keystore and truststore of that one node
> and make sure it has what you would expect, and also double check
> nifi.properties compared to the others to see if anything seems different.
>
>
>
> Changing all of the keystores, truststores, etc should be fine from a data
> perspective...
>
>
>
> If you decide to go that route it would probably be easiest to start back
> over from a security perspective, meaning:
>
> - Stop all the nodes and delete the users.xml and authorizations.xml from
> all nodes
>
> - Configure authorizers.xml with the appropriate initial admin (or legacy
> file) and node identities based on the new certs
>
> - Ensure authorizers.xml is the same on all nodes
>
> - Then restart everything
>
>
>
> Alternatively, you might be able to manually add users for all of the new
> certs before shutting everything down and give them the appropriate
> policies, then restart everything, but requires more manual work to get
> everything lined up.
>
>
>
>
>
> On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton
> <co...@secdata.com> wrote:
>
> Hi,
>
> As a plan for tomorrow – I have generated new keystores, truststores, client
> certts etc. for all nodes in my cluster using the
>
>
>
> From: Bryan Bende <bb...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Wednesday, 19 October 2016 at 15:33
>
>
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
>
>
> Trying to think of things to check here...
>
>
>
> Does every node have nifi.remote.input.secure=true in nifi.properties and
> the URL in the RPG is an https URL?
>
>
>
> On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton
> <co...@secdata.com> wrote:
>
> One other thing…
>
> The RPGs have an unlocked padlock on them saying S2S is not secure.
>
> Conrad
>
>
>
> From: Bryan Bende <bb...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Wednesday, 19 October 2016 at 15:20
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Ok that does seem like a TLS/SSL issue...
>
>
>
> Is this a single cluster doing site-to-site to itself?
>
>
>
> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com> wrote:
>
> thanks conrad - did get it. Bryan is being more helpful that I so I
> went silent :-)
>
> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>
> <co...@secdata.com> wrote:
>> Hi Joe,
>> Yep,
>> Tried removing the RPG that referenced the NCM and adding new one with
>> one of the nifis as url.
>> That sort of worked, but kept getting errors about the NCM not being
>> available for the ports and therefore couldn’t actually enable the port I
>> needed to for that RPG.
>> Thanks
>> Conrad
>>
>> (sending again as don’t know if the stupid header ‘spoofed’ is stopping
>> getting though – apologies if already sent)
>>
>> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
>>
>> Conrad,
>>
>> For s2s now you can just point at any of the nodes in the cluster.
>> Have you tried changing the URL or removing and adding new RPG
>> entries?
>>
>> Thanks
>> Joe
>>
>> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
>> <co...@secdata.com> wrote:
>> > Hi,
>> >
>> > I have finally taken the plunge to upgrade my cluster from 0.6.1
>> to 1.0.0.
>> >
>> > 6 nodes with a NCM.
>> >
>> > With the removal of NCM in 1.0.0 I believe I now have an issue
>> where none of
>> > my Remote Process Groups work as they previously did because
>> they were
>> > configured to connect to the NCM (as the RPG url) which now
>> doesn’t exist.
>> >
>> > I have tried converting my NCM to a node but whilst I can get it
>> running
>> > (sort of) when I try and connect to the cluster I get something
>> like this in
>> > my logs…
>> >
>> >
>> >
>> > 2016-10-19 13:14:44,109 ERROR [main]
>> o.a.nifi.controller.StandardFlowService
>> > Failed to load flow from cluster due to:
>> > org.apache.nifi.controller.UninheritableFlowException: Failed to
>> connect
>> > node to cluster because local flow is different than cluster
>> flow.
>> >
>> > org.apache.nifi.controller.UninheritableFlowException: Failed to
>> connect
>> > node to cluster because local flow is different than cluster
>> flow.
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> > [nifi-jetty-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > Caused by:
>> org.apache.nifi.controller.UninheritableFlowException: Proposed
>> > Authorizer is not inheritable by the flow controller because of
>> Authorizer
>> > differences: Proposed Authorizations do not match current
>> Authorizations
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > ... 4 common frames omitted
>> >
>> > 2016-10-19 13:14:44,414 ERROR [main]
>> o.a.n.c.c.node.NodeClusterCoordinator
>> > Event Reported for ncm-cm1.local:9090 -- Node disconnected from
>> > cluster due to
>> org.apache.nifi.controller.UninheritableFlowException: Failed
>> > to connect node to cluster because local flow is different than
>> cluster
>> > flow.
>> >
>> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
>> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
>> Thread[Shutdown
>> > Cluster Coordinator,5,main]: java.lang.NullPointerException
>> >
>> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
>> > org.apache.nifi.NiFi
>> >
>> > java.lang.NullPointerException: null
>> >
>> > at
>> >
>> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
>> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
>> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
>> >
>> > 2016-10-19 13:14:44,448 WARN [main]
>> o.a.n.c.l.e.CuratorLeaderElectionManager
>> > Failed to close Leader Selector for Cluster Coordinator
>> >
>> > java.lang.IllegalStateException: Already closed or has not been
>> started
>> >
>> > at
>> >
>> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
>> > ~[guava-18.0.jar:na]
>> >
>> > at
>> >
>> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
>> > ~[curator-recipes-2.11.0.jar:na]
>> >
>> > at
>> >
>> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
>> > [nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
>> > [nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
>> > [nifi-jetty-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
>> > org.apache.nifi.io.socket.SocketListener Failed to communicate
>> with Unknown
>> > Host due to java.net.SocketException: Socket closed
>> >
>> > java.net.SocketException: Socket closed
>> >
>> > at java.net.PlainSocketImpl.socketAccept(Native Method)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
>> > ~[na:1.8.0_51]
>> >
>> > at
>> >
>> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
>> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>> >
>> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
>> >
>> > 2016-10-19 13:14:45,064 WARN [main]
>> org.apache.nifi.web.server.JettyServer
>> > Failed to start web server... shutting down.
>> >
>> > java.lang.Exception: Unable to load flow due to:
>> java.io.IOException:
>> > org.apache.nifi.controller.UninheritableFlowException: Failed to
>> connect
>> > node to cluster because local flow is different than cluster
>> flow.
>> >
>> > at
>> >
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
>> > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> > [nifi-runtime-1.0.0.jar:1.0.0]
>> >
>> > Caused by: java.io.IOException:
>> > org.apache.nifi.controller.UninheritableFlowException: Failed to
>> connect
>> > node to cluster because local flow is different than cluster
>> flow.
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> >
>> > ... 2 common frames omitted
>> >
>> > Caused by:
>> org.apache.nifi.controller.UninheritableFlowException: Failed to
>> > connect node to cluster because local flow is different than
>> cluster flow.
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > ... 3 common frames omitted
>> >
>> > Caused by:
>> org.apache.nifi.controller.UninheritableFlowException: Proposed
>> > Authorizer is not inheritable by the flow controller because of
>> Authorizer
>> > differences: Proposed Authorizations do not match current
>> Authorizations
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > at
>> >
>> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> >
>> > ... 4 common frames omitted
>> >
>> > [root@ncm-cm1 logs]#
>> >
>> >
>> >
>> > I don’t know if the ‘Proposed Authorizer is not inheritable…’
>> exception is
>> > part of the problem too.
>> >
>> > The docs weren’t very clear on whether (when upgrading and using
>> the legacy
>> > support of the authorized-user.xml path required the nodes to be
>> also added
>> > to the authorizers.xml.
>> >
>> > I did add them in the end as various attempts to get the cluster
>> up and
>> > running without them failed (as each server didn’t seem to have
>> rights to do
>> > anything.
>> >
>> >
>> >
>> > I have a lot of RPG in my work flows as I am ingesting many
>> syslog data
>> > sources and this was the recommended pattern to distribute the
>> data
>> > (listensyslog…run on primary, output to port (RPG), pick up in
>> rest of data
>> > flow),
>> >
>> >
>> >
>> > Any suggestions on where to start trying to get this working?
>> >
>> > I’ve tried creating a new RPG on one on the nifis and connecting
>> the
>> > syslog to that which sort of worked but then I have a bunch of
>> other errors
>> > when trying to enable the ports to do with not being able to
>> connect to
>> > (what was) the NCM.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Conrad
>> >
>> >
>> >
>> > SecureData, combating cyber threats
>> >
>> > ________________________________
>> >
>> > The information contained in this message or any of its
>> attachments may be
>> > privileged and confidential and intended for the exclusive use
>> of the
>> > intended recipient. If you are not the intended recipient any
>> disclosure,
>> > reproduction, distribution or other dissemination or use of this
>> > communications is strictly prohibited. The views expressed in
>> this email are
>> > those of the individual and not necessarily of SecureData Europe
>> Ltd. Any
>> > prices quoted are only valid if followed up by a formal written
>> quote.
>> >
>> > SecureData Europe Limited. Registered in England & Wales
>> 04365896.
>> > Registered Address: SecureData House, Hermitage Court, Hermitage
>> Lane,
>> > Maidstone, Kent, ME16 9NT
>>
>>
>> ***This email originated outside SecureData***
>>
>> Click
>> https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==
>> to report this email as spam.
>>
>>
>>
>>
>
>
>
>
>
>
>
>
>
> * Unknown Key
> * 0x2F7DEF69
>
>
>
>
>
> * Unknown Key
> * 0x2F7DEF69
>
>
RE: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Hi,
I realise this may be getting boring for most but…
I didn’t find any resolution to the upgrade so I have cleanly installed v1.0.0 and oddly experienced the same issue with RPGs in that although the RPGs didn’t show any errors (in so much as they had no warning on them and reported that S2S was secure) the errors reported were about not being able to determine other nodes in cluster.
This is a section of log that showed an error that I don’t think I saw before but only including here in case one of you fine folks need it for any reason…
ERROR [Site-to-Site Worker Thread-194] o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote instance Peer[url=nifi://nifinode3-cm1.mis.local:57289] (SocketFlowFileServerProtocol[CommsID=04674c10-7351-443f-99c8-bffc59d650a7]) due to java.lang.NullPointerException; closing connection
2016-10-21 12:19:11,401 ERROR [Site-to-Site Worker Thread-195] o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote instance Peer[url=nifi://nifinode1-cm1.mis.local:35831] (SocketFlowFileServerProtocol[CommsID=97eb2f1c-f3dd-4924-89c9-d294bb4037f5]) due to java.lang.NullPointerException; closing connection
2016-10-21 12:19:11,455 ERROR [Site-to-Site Worker Thread-196] o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote instance Peer[url=nifi://nifinode5-cm1.mis.local:59093] (SocketFlowFileServerProtocol[CommsID=61e129ca-2e21-477a-8201-16b905e5beb6]) due to java.lang.NullPointerException; closing connection
2016-10-21 12:19:11,462 ERROR [Site-to-Site Worker Thread-197] o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote instance Peer[url=nifi://nifinode6-cm1.mis.local:37275] (SocketFlowFileServerProtocol[CommsID=48ec62f4-a9ba-45a7-a149-4892d0193819]) due to java.lang.NullPointerException; closing connection
2016-10-21 12:19:11,470 ERROR [Site-to-Site Worker Thread-198] o.a.nifi.remote.SocketRemoteSiteListener Unable to communicate with remote instance Peer[url=nifi://nifinode4-cm1.mis.local:57745] (SocketFlowFileServerProtocol[CommsID=266459a8-7a9b-44bd-8047-b5ba4d9b67ec]) due to java.lang.NullPointerException; closing connection
in my naivety this suggests a problem with the socket (RAW) connection protocol. The relevant section for S2S connection in my nifi.properties is
nifi.remote.input.socket.host=nifinode6-cm1.mis.local // the number different for each node obviously
nifi.remote.input.socket.port=10443
nifi.remote.input.secure=true
nifi.remote.input.http.enabled=false
nifi.remote.input.http.transaction.ttl=30 sec
the errors suggest that the port specified here aren’t being used, but some random ports are being used instead. Of course this is complete supposition and probably a red herring.
Anyway, I updated my nifi.properties to
nifi.remote.input.socket.host=nifinode6-cm1.mis.local
nifi.remote.input.http.host=nifinode6-cm1.mis.local
nifi.remote.input.socket.port=10443
nifi.remote.input.http.port=11443
nifi.remote.input.secure=true
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
and used HTTP for my RPG and is now working ok.
The test harness for this is
GenerateFlowFile->RGP(test input port)
InputPort(test input)->LogAttribute
Regards,
Conrad
From: Conrad Crampton <co...@SecData.com>
Date: Friday, 21 October 2016 at 08:18
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Hi,
Yes, the access policy for the input port at the root level of my workspace has S2S access policy (receive data via site to site) for all server nodes (I have all my nodes in one group).
For the next level down in my workspace (I have other process groups from my main ‘root’ page in the UI space to organise the flows), I have input ports on the next level down of flows which when I check the access policies on that for S2S, the receive and send data via site-to-site options are greyed out and if I try to override the policy, they still are. I don’t know if this is important, but from reading the post below, the issue that the access policies looks to address is different from my issue. The RPG has a locked padlock and says site to site is secure. I don’t have any warning triangles on the RPG itself, but I have the aforementioned warnings in my logs.
I’m going to abandon this now as I can’t get it to work.
I’m going to try a fresh install and go with that – and have to recreate all my flows where necessary. I’m moving to a new model of data ingestion anyway so isn’t as catastrophic as it might be.
Thanks for the help.
Conrad
From: Andy LoPresto <al...@apache.org>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, 20 October 2016 at 17:31
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
* PGP Signed by an unknown key
Conrad,
For the site-to-site did you follow the instructions here [1]? Each node needs to be added as a user in order to make the connections.
[1] http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Oct 20, 2016, at 7:36 AM, Conrad Crampton <co...@secdata.com>> wrote:
Ok,
So I tried everything suggested so far to no avail unfortunately.
So what I have done is to create all new certs etc. using the tookit. Updated my existing authoriszed-users.xml to have to match the full cert distinguished names CN=server.name, OU=NIFI etc.
Recreated all my remote process groups to not reference the original NCM as that still wouldn’t work – after a complete new install (upgrade).
So now what I have is a six node cluster using original data/worker nodes and they are part of the cluster – all appears to be working ie. I can log into the UI (nice by the way ;-) on each server. There are no SSL handshake errors etc. BUT the RPG (newly created) still don’t appear to be working. I am getting
11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
I can telnet from server to server on both https (UI) port and S2S port.
I am really at a loss as to what to do now.
Data is queuing up in my input processors with nowhere to go.
Do I have to do something radical here to get this working like stopping everything, clearing out all the queues then starting up again??? I really don’t want to do this obviously but I am getting nowhere on this – two days of frustration with nothing to show for it.
Any more suggestions please??
Thanks for your patience.
Conrad
From: Andy LoPresto <al...@apache.org>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 18:24
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> Old Signed by an unknown key
Hi Conrad,
Bryan is correct that changing the certificates (and the encapsulating keystores and truststores) will not affect any data held in the nodes.
Regenerating everything using the TLS toolkit should hopefully not be too challenging, but I am also curious as to why you are getting these handshake exceptions now. As Bryan pointed out, adding the following line to bootstrap.conf will provide substantial additional log output which should help trace the issue.
java.arg.15=-Djavax.net.debug=ssl,handshake
You can also imitate the node connecting to the (previous) NCM via this command:
$ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
Where:
* <host:port> = the hostname and port of the “NCM”
* <path_to_your_cert.pem> = the public key used to identify the “node” (can be exported from the node keystore [1])
* <path_to_your_key.pem> = the private key used to identify the “node” (can be exported from the node keystore via 2 step process)
* <path_to_your_CA_cert.pem> = the public key used to sign the “NCM” certificate (could be a 3rd party like Verisign or DigiCert, or an internal organization CA if you have one)
If you’ve already regenerated everything and it works, that’s fine. But if you have the time to try and investigate the old certs, we are interested and prepared to help. Thanks.
[1] https://security.stackexchange.com/a/66865/16485
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com>> wrote:
That is definitely weird that it is only an issue on the node that used to be the NCM.
Might be worth double checking the keystore and truststore of that one node and make sure it has what you would expect, and also double check nifi.properties compared to the others to see if anything seems different.
Changing all of the keystores, truststores, etc should be fine from a data perspective...
If you decide to go that route it would probably be easiest to start back over from a security perspective, meaning:
- Stop all the nodes and delete the users.xml and authorizations.xml from all nodes
- Configure authorizers.xml with the appropriate initial admin (or legacy file) and node identities based on the new certs
- Ensure authorizers.xml is the same on all nodes
- Then restart everything
Alternatively, you might be able to manually add users for all of the new certs before shutting everything down and give them the appropriate policies, then restart everything, but requires more manual work to get everything lined up.
On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton <co...@secdata.com>> wrote:
Hi,
As a plan for tomorrow – I have generated new keystores, truststores, client certts etc. for all nodes in my cluster using the
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:33
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Trying to think of things to check here...
Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL?
On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <co...@secdata.com>> wrote:
One other thing…
The RPGs have an unlocked padlock on them saying S2S is not secure.
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:20
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Ok that does seem like a TLS/SSL issue...
Is this a single cluster doing site-to-site to itself?
On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>> wrote:
thanks conrad - did get it. Bryan is being more helpful that I so I
went silent :-)
On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
<co...@secdata.com>> wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with one of the nifis as url.
> That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com>> wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com>> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net<http://java.net/>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > connect node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the legacy
> > support of the authorized-user.xml path required the nodes to be also added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the nifis and connecting the
> > syslog to that which sort of worked but then I have a bunch of other errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
>
> ***This email originated outside SecureData***
>
> Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
>
>
>
>
* Unknown Key
* 0x2F7DEF69
* Unknown Key
* 0x2F7DEF69
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Hi,
Yes, the access policy for the input port at the root level of my workspace has S2S access policy (receive data via site to site) for all server nodes (I have all my nodes in one group).
For the next level down in my workspace (I have other process groups from my main ‘root’ page in the UI space to organise the flows), I have input ports on the next level down of flows which when I check the access policies on that for S2S, the receive and send data via site-to-site options are greyed out and if I try to override the policy, they still are. I don’t know if this is important, but from reading the post below, the issue that the access policies looks to address is different from my issue. The RPG has a locked padlock and says site to site is secure. I don’t have any warning triangles on the RPG itself, but I have the aforementioned warnings in my logs.
I’m going to abandon this now as I can’t get it to work.
I’m going to try a fresh install and go with that – and have to recreate all my flows where necessary. I’m moving to a new model of data ingestion anyway so isn’t as catastrophic as it might be.
Thanks for the help.
Conrad
From: Andy LoPresto <al...@apache.org>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, 20 October 2016 at 17:31
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
* PGP Signed by an unknown key
Conrad,
For the site-to-site did you follow the instructions here [1]? Each node needs to be added as a user in order to make the connections.
[1] http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Oct 20, 2016, at 7:36 AM, Conrad Crampton <co...@secdata.com>> wrote:
Ok,
So I tried everything suggested so far to no avail unfortunately.
So what I have done is to create all new certs etc. using the tookit. Updated my existing authoriszed-users.xml to have to match the full cert distinguished names CN=server.name, OU=NIFI etc.
Recreated all my remote process groups to not reference the original NCM as that still wouldn’t work – after a complete new install (upgrade).
So now what I have is a six node cluster using original data/worker nodes and they are part of the cluster – all appears to be working ie. I can log into the UI (nice by the way ;-) on each server. There are no SSL handshake errors etc. BUT the RPG (newly created) still don’t appear to be working. I am getting
11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
I can telnet from server to server on both https (UI) port and S2S port.
I am really at a loss as to what to do now.
Data is queuing up in my input processors with nowhere to go.
Do I have to do something radical here to get this working like stopping everything, clearing out all the queues then starting up again??? I really don’t want to do this obviously but I am getting nowhere on this – two days of frustration with nothing to show for it.
Any more suggestions please??
Thanks for your patience.
Conrad
From: Andy LoPresto <al...@apache.org>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 18:24
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
> Old Signed by an unknown key
Hi Conrad,
Bryan is correct that changing the certificates (and the encapsulating keystores and truststores) will not affect any data held in the nodes.
Regenerating everything using the TLS toolkit should hopefully not be too challenging, but I am also curious as to why you are getting these handshake exceptions now. As Bryan pointed out, adding the following line to bootstrap.conf will provide substantial additional log output which should help trace the issue.
java.arg.15=-Djavax.net.debug=ssl,handshake
You can also imitate the node connecting to the (previous) NCM via this command:
$ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
Where:
* <host:port> = the hostname and port of the “NCM”
* <path_to_your_cert.pem> = the public key used to identify the “node” (can be exported from the node keystore [1])
* <path_to_your_key.pem> = the private key used to identify the “node” (can be exported from the node keystore via 2 step process)
* <path_to_your_CA_cert.pem> = the public key used to sign the “NCM” certificate (could be a 3rd party like Verisign or DigiCert, or an internal organization CA if you have one)
If you’ve already regenerated everything and it works, that’s fine. But if you have the time to try and investigate the old certs, we are interested and prepared to help. Thanks.
[1] https://security.stackexchange.com/a/66865/16485
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com>> wrote:
That is definitely weird that it is only an issue on the node that used to be the NCM.
Might be worth double checking the keystore and truststore of that one node and make sure it has what you would expect, and also double check nifi.properties compared to the others to see if anything seems different.
Changing all of the keystores, truststores, etc should be fine from a data perspective...
If you decide to go that route it would probably be easiest to start back over from a security perspective, meaning:
- Stop all the nodes and delete the users.xml and authorizations.xml from all nodes
- Configure authorizers.xml with the appropriate initial admin (or legacy file) and node identities based on the new certs
- Ensure authorizers.xml is the same on all nodes
- Then restart everything
Alternatively, you might be able to manually add users for all of the new certs before shutting everything down and give them the appropriate policies, then restart everything, but requires more manual work to get everything lined up.
On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton <co...@secdata.com>> wrote:
Hi,
As a plan for tomorrow – I have generated new keystores, truststores, client certts etc. for all nodes in my cluster using the
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:33
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Trying to think of things to check here...
Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL?
On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <co...@secdata.com>> wrote:
One other thing…
The RPGs have an unlocked padlock on them saying S2S is not secure.
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:20
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Ok that does seem like a TLS/SSL issue...
Is this a single cluster doing site-to-site to itself?
On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>> wrote:
thanks conrad - did get it. Bryan is being more helpful that I so I
went silent :-)
On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
<co...@secdata.com>> wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with one of the nifis as url.
> That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com>> wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com>> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net<http://java.net/>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > connect node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the legacy
> > support of the authorized-user.xml path required the nodes to be also added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the nifis and connecting the
> > syslog to that which sort of worked but then I have a bunch of other errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
>
> ***This email originated outside SecureData***
>
> Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
>
>
>
>
* Unknown Key
* 0x2F7DEF69
* Unknown Key
* 0x2F7DEF69
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Andy LoPresto <al...@apache.org>.
Conrad,
For the site-to-site did you follow the instructions here [1]? Each node needs to be added as a user in order to make the connections.
[1] http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site <http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Oct 20, 2016, at 7:36 AM, Conrad Crampton <co...@SecData.com> wrote:
>
> Ok,
> So I tried everything suggested so far to no avail unfortunately.
>
> So what I have done is to create all new certs etc. using the tookit. Updated my existing authoriszed-users.xml to have to match the full cert distinguished names CN=server.name, OU=NIFI etc.
>
> Recreated all my remote process groups to not reference the original NCM as that still wouldn’t work – after a complete new install (upgrade).
>
> So now what I have is a six node cluster using original data/worker nodes and they are part of the cluster – all appears to be working ie. I can log into the UI (nice by the way ;-) on each server. There are no SSL handshake errors etc. BUT the RPG (newly created) still don’t appear to be working. I am getting
>
> 11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
> 11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
> 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
> 11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
> 11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
> nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
> 11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
> 11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
> nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
>
> I can telnet from server to server on both https (UI) port and S2S port.
> I am really at a loss as to what to do now.
>
> Data is queuing up in my input processors with nowhere to go.
> Do I have to do something radical here to get this working like stopping everything, clearing out all the queues then starting up again??? I really don’t want to do this obviously but I am getting nowhere on this – two days of frustration with nothing to show for it.
>
> Any more suggestions please??
> Thanks for your patience.
> Conrad
>
>
> From: Andy LoPresto <alopresto@apache.org <ma...@apache.org>>
> Reply-To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Date: Wednesday, 19 October 2016 at 18:24
> To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
> * PGP Signed by an unknown key
> Hi Conrad,
>
> Bryan is correct that changing the certificates (and the encapsulating keystores and truststores) will not affect any data held in the nodes.
>
> Regenerating everything using the TLS toolkit should hopefully not be too challenging, but I am also curious as to why you are getting these handshake exceptions now. As Bryan pointed out, adding the following line to bootstrap.conf will provide substantial additional log output which should help trace the issue.
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> You can also imitate the node connecting to the (previous) NCM via this command:
>
> $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
>
> Where:
>
> <host:port> = the hostname and port of the “NCM”
> <path_to_your_cert.pem> = the public key used to identify the “node” (can be exported from the node keystore [1])
> <path_to_your_key.pem> = the private key used to identify the “node” (can be exported from the node keystore via 2 step process)
> <path_to_your_CA_cert.pem> = the public key used to sign the “NCM” certificate (could be a 3rd party like Verisign or DigiCert, or an internal organization CA if you have one)
>
> If you’ve already regenerated everything and it works, that’s fine. But if you have the time to try and investigate the old certs, we are interested and prepared to help. Thanks.
>
> [1] https://security.stackexchange.com/a/66865/16485 <https://security.stackexchange.com/a/66865/16485>
>
> Andy LoPresto
> alopresto@apache.org <ma...@apache.org>
> alopresto.apache@gmail.com <ma...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Oct 19, 2016, at 1:03 PM, Bryan Bende <bbende@gmail.com <ma...@gmail.com>> wrote:
>
> That is definitely weird that it is only an issue on the node that used to be the NCM.
> Might be worth double checking the keystore and truststore of that one node and make sure it has what you would expect, and also double check nifi.properties compared to the others to see if anything seems different.
>
> Changing all of the keystores, truststores, etc should be fine from a data perspective...
>
> If you decide to go that route it would probably be easiest to start back over from a security perspective, meaning:
> - Stop all the nodes and delete the users.xml and authorizations.xml from all nodes
> - Configure authorizers.xml with the appropriate initial admin (or legacy file) and node identities based on the new certs
> - Ensure authorizers.xml is the same on all nodes
> - Then restart everything
>
> Alternatively, you might be able to manually add users for all of the new certs before shutting everything down and give them the appropriate policies, then restart everything, but requires more manual work to get everything lined up.
>
>
> On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
> Hi,
> As a plan for tomorrow – I have generated new keystores, truststores, client certts etc. for all nodes in my cluster using the
>
> From: Bryan Bende <bbende@gmail.com <ma...@gmail.com>>
> Reply-To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Date: Wednesday, 19 October 2016 at 15:33
>
> To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
> Trying to think of things to check here...
>
> Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL?
>
> On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
> One other thing…
> The RPGs have an unlocked padlock on them saying S2S is not secure.
> Conrad
>
> From: Bryan Bende <bbende@gmail.com <ma...@gmail.com>>
> Reply-To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Date: Wednesday, 19 October 2016 at 15:20
> To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
> Ok that does seem like a TLS/SSL issue...
>
> Is this a single cluster doing site-to-site to itself?
>
> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
> thanks conrad - did get it. Bryan is being more helpful that I so I
> went silent :-)
>
> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
> <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
> > Hi Joe,
> > Yep,
> > Tried removing the RPG that referenced the NCM and adding new one with one of the nifis as url.
> > That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> > Thanks
> > Conrad
> >
> > (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
> >
> > On 19/10/2016, 14:12, "Joe Witt" <joe.witt@gmail.com <ma...@gmail.com>> wrote:
> >
> > Conrad,
> >
> > For s2s now you can just point at any of the nodes in the cluster.
> > Have you tried changing the URL or removing and adding new RPG
> > entries?
> >
> > Thanks
> > Joe
> >
> > On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> > <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
> > > Hi,
> > >
> > > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> > >
> > > 6 nodes with a NCM.
> > >
> > > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > > my Remote Process Groups work as they previously did because they were
> > > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> > >
> > > I have tried converting my NCM to a node but whilst I can get it running
> > > (sort of) when I try and connect to the cluster I get something like this in
> > > my logs…
> > >
> > >
> > >
> > > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > > Failed to load flow from cluster due to:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > > node to cluster because local flow is different than cluster flow.
> > >
> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > > node to cluster because local flow is different than cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > > Authorizer is not inheritable by the flow controller because of Authorizer
> > > differences: Proposed Authorizations do not match current Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > > Event Reported for ncm-cm1.local:9090 -- Node disconnected from
> > > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > > to connect node to cluster because local flow is different than cluster
> > > flow.
> > >
> > > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > > Cluster Coordinator,5,main]: java.lang.NullPointerException
> > >
> > > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi
> > >
> > > java.lang.NullPointerException: null
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > > Failed to close Leader Selector for Cluster Coordinator
> > >
> > > java.lang.IllegalStateException: Already closed or has not been started
> > >
> > > at
> > > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > > ~[guava-18.0.jar:na]
> > >
> > > at
> > > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > > ~[curator-recipes-2.11.0.jar:na]
> > >
> > > at
> > > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > > Host due to java.net.SocketException: Socket closed
> > >
> > > java.net.SocketException: Socket closed
> > >
> > > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.net <http://java.net/>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > > ~[na:1.8.0_51]
> > >
> > > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > > Failed to start web server... shutting down.
> > >
> > > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > > node to cluster because local flow is different than cluster flow.
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > > node to cluster because local flow is different than cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > ... 2 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > > connect node to cluster because local flow is different than cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 3 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > > Authorizer is not inheritable by the flow controller because of Authorizer
> > > differences: Proposed Authorizations do not match current Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > [root@ncm-cm1 logs]#
> > >
> > >
> > >
> > > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > > part of the problem too.
> > >
> > > The docs weren’t very clear on whether (when upgrading and using the legacy
> > > support of the authorized-user.xml path required the nodes to be also added
> > > to the authorizers.xml.
> > >
> > > I did add them in the end as various attempts to get the cluster up and
> > > running without them failed (as each server didn’t seem to have rights to do
> > > anything.
> > >
> > >
> > >
> > > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > > sources and this was the recommended pattern to distribute the data
> > > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > > flow),
> > >
> > >
> > >
> > > Any suggestions on where to start trying to get this working?
> > >
> > > I’ve tried creating a new RPG on one on the nifis and connecting the
> > > syslog to that which sort of worked but then I have a bunch of other errors
> > > when trying to enable the ports to do with not being able to connect to
> > > (what was) the NCM.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Conrad
> > >
> > >
> > >
> > > SecureData, combating cyber threats
> > >
> > > ________________________________
> > >
> > > The information contained in this message or any of its attachments may be
> > > privileged and confidential and intended for the exclusive use of the
> > > intended recipient. If you are not the intended recipient any disclosure,
> > > reproduction, distribution or other dissemination or use of this
> > > communications is strictly prohibited. The views expressed in this email are
> > > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > > prices quoted are only valid if followed up by a formal written quote.
> > >
> > > SecureData Europe Limited. Registered in England & Wales 04365896.
> > > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > > Maidstone, Kent, ME16 9NT
> >
> >
> > ***This email originated outside SecureData***
> >
> > Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A== <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
> >
> >
> >
> >
>
>
>
>
> * Unknown Key
> * 0x2F7DEF69
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Ok,
So I tried everything suggested so far to no avail unfortunately.
So what I have done is to create all new certs etc. using the tookit. Updated my existing authoriszed-users.xml to have to match the full cert distinguished names CN=server.name, OU=NIFI etc.
Recreated all my remote process groups to not reference the original NCM as that still wouldn’t work – after a complete new install (upgrade).
So now what I have is a six node cluster using original data/worker nodes and they are part of the cluster – all appears to be working ie. I can log into the UI (nice by the way ;-) on each server. There are no SSL handshake errors etc. BUT the RPG (newly created) still don’t appear to be working. I am getting
11:34:24 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@782af623 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:25 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
nifi1-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3a547274 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@54c2df1 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:25 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi5-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@50d59f3c Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:26 GMTWARNINGe19ccf8e-0157-1000-0000-000063bfd9c0
nifi2-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@97c92ef Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:26 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi6-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@70663037 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
11:34:27 GMTWARNINGe1990203-0157-1000-ffff-ffff9ff40dc0
nifi4-cm1.local:9443org.apache.nifi.remote.client.PeerSelector@3c040426 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster
I can telnet from server to server on both https (UI) port and S2S port.
I am really at a loss as to what to do now.
Data is queuing up in my input processors with nowhere to go.
Do I have to do something radical here to get this working like stopping everything, clearing out all the queues then starting up again??? I really don’t want to do this obviously but I am getting nowhere on this – two days of frustration with nothing to show for it.
Any more suggestions please??
Thanks for your patience.
Conrad
From: Andy LoPresto <al...@apache.org>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 19 October 2016 at 18:24
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
* PGP Signed by an unknown key
Hi Conrad,
Bryan is correct that changing the certificates (and the encapsulating keystores and truststores) will not affect any data held in the nodes.
Regenerating everything using the TLS toolkit should hopefully not be too challenging, but I am also curious as to why you are getting these handshake exceptions now. As Bryan pointed out, adding the following line to bootstrap.conf will provide substantial additional log output which should help trace the issue.
java.arg.15=-Djavax.net.debug=ssl,handshake
You can also imitate the node connecting to the (previous) NCM via this command:
$ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
Where:
* <host:port> = the hostname and port of the “NCM”
* <path_to_your_cert.pem> = the public key used to identify the “node” (can be exported from the node keystore [1])
* <path_to_your_key.pem> = the private key used to identify the “node” (can be exported from the node keystore via 2 step process)
* <path_to_your_CA_cert.pem> = the public key used to sign the “NCM” certificate (could be a 3rd party like Verisign or DigiCert, or an internal organization CA if you have one)
If you’ve already regenerated everything and it works, that’s fine. But if you have the time to try and investigate the old certs, we are interested and prepared to help. Thanks.
[1] https://security.stackexchange.com/a/66865/16485
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com>> wrote:
That is definitely weird that it is only an issue on the node that used to be the NCM.
Might be worth double checking the keystore and truststore of that one node and make sure it has what you would expect, and also double check nifi.properties compared to the others to see if anything seems different.
Changing all of the keystores, truststores, etc should be fine from a data perspective...
If you decide to go that route it would probably be easiest to start back over from a security perspective, meaning:
- Stop all the nodes and delete the users.xml and authorizations.xml from all nodes
- Configure authorizers.xml with the appropriate initial admin (or legacy file) and node identities based on the new certs
- Ensure authorizers.xml is the same on all nodes
- Then restart everything
Alternatively, you might be able to manually add users for all of the new certs before shutting everything down and give them the appropriate policies, then restart everything, but requires more manual work to get everything lined up.
On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton <co...@secdata.com>> wrote:
Hi,
As a plan for tomorrow – I have generated new keystores, truststores, client certts etc. for all nodes in my cluster using the
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:33
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Trying to think of things to check here...
Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL?
On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <co...@secdata.com>> wrote:
One other thing…
The RPGs have an unlocked padlock on them saying S2S is not secure.
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:20
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Ok that does seem like a TLS/SSL issue...
Is this a single cluster doing site-to-site to itself?
On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>> wrote:
thanks conrad - did get it. Bryan is being more helpful that I so I
went silent :-)
On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
<co...@secdata.com>> wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with one of the nifis as url.
> That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com>> wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com>> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net<http://java.net/>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > connect node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the legacy
> > support of the authorized-user.xml path required the nodes to be also added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the nifis and connecting the
> > syslog to that which sort of worked but then I have a bunch of other errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
>
> ***This email originated outside SecureData***
>
> Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
>
>
>
>
* Unknown Key
* 0x2F7DEF69
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Andy LoPresto <al...@apache.org>.
Also worth noting that there were some changes to how cluster and site-to-site communications worked with certificates between 0.6.1 and 1.0.0:
* https://issues.apache.org/jira/browse/NIFI-1981 <https://issues.apache.org/jira/browse/NIFI-1981> (0.6.1 -> 0.7.0)
* https://issues.apache.org/jira/browse/NIFI-1753 <https://issues.apache.org/jira/browse/NIFI-1753>
* https://issues.apache.org/jira/browse/NIFI-1857 <https://issues.apache.org/jira/browse/NIFI-1857>
* https://issues.apache.org/jira/browse/NIFI-2186 <https://issues.apache.org/jira/browse/NIFI-2186>
Future enhancements:
* https://issues.apache.org/jira/browse/NIFI-1990 <https://issues.apache.org/jira/browse/NIFI-1990>
* https://issues.apache.org/jira/browse/NIFI-1995 <https://issues.apache.org/jira/browse/NIFI-1995>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Oct 19, 2016, at 1:24 PM, Andy LoPresto <al...@apache.org> wrote:
>
> Hi Conrad,
>
> Bryan is correct that changing the certificates (and the encapsulating keystores and truststores) will not affect any data held in the nodes.
>
> Regenerating everything using the TLS toolkit should hopefully not be too challenging, but I am also curious as to why you are getting these handshake exceptions now. As Bryan pointed out, adding the following line to bootstrap.conf will provide substantial additional log output which should help trace the issue.
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> You can also imitate the node connecting to the (previous) NCM via this command:
>
> $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
>
> Where:
>
> <host:port> = the hostname and port of the “NCM”
> <path_to_your_cert.pem> = the public key used to identify the “node” (can be exported from the node keystore [1])
> <path_to_your_key.pem> = the private key used to identify the “node” (can be exported from the node keystore via 2 step process)
> <path_to_your_CA_cert.pem> = the public key used to sign the “NCM” certificate (could be a 3rd party like Verisign or DigiCert, or an internal organization CA if you have one)
>
> If you’ve already regenerated everything and it works, that’s fine. But if you have the time to try and investigate the old certs, we are interested and prepared to help. Thanks.
>
> [1] https://security.stackexchange.com/a/66865/16485 <https://security.stackexchange.com/a/66865/16485>
>
> Andy LoPresto
> alopresto@apache.org <ma...@apache.org>
> alopresto.apache@gmail.com <ma...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Oct 19, 2016, at 1:03 PM, Bryan Bende <bbende@gmail.com <ma...@gmail.com>> wrote:
>>
>> That is definitely weird that it is only an issue on the node that used to be the NCM.
>> Might be worth double checking the keystore and truststore of that one node and make sure it has what you would expect, and also double check nifi.properties compared to the others to see if anything seems different.
>>
>> Changing all of the keystores, truststores, etc should be fine from a data perspective...
>>
>> If you decide to go that route it would probably be easiest to start back over from a security perspective, meaning:
>> - Stop all the nodes and delete the users.xml and authorizations.xml from all nodes
>> - Configure authorizers.xml with the appropriate initial admin (or legacy file) and node identities based on the new certs
>> - Ensure authorizers.xml is the same on all nodes
>> - Then restart everything
>>
>> Alternatively, you might be able to manually add users for all of the new certs before shutting everything down and give them the appropriate policies, then restart everything, but requires more manual work to get everything lined up.
>>
>>
>> On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
>> Hi,
>>
>> As a plan for tomorrow – I have generated new keystores, truststores, client certts etc. for all nodes in my cluster using the
>>
>>
>>
>> From: Bryan Bende <bbende@gmail.com <ma...@gmail.com>>
>> Reply-To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
>> Date: Wednesday, 19 October 2016 at 15:33
>>
>>
>> To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
>> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>>
>>
>>
>> Trying to think of things to check here...
>>
>>
>>
>> Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL?
>>
>>
>>
>> On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
>>
>> One other thing…
>>
>> The RPGs have an unlocked padlock on them saying S2S is not secure.
>>
>> Conrad
>>
>>
>>
>> From: Bryan Bende <bbende@gmail.com <ma...@gmail.com>>
>> Reply-To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
>> Date: Wednesday, 19 October 2016 at 15:20
>> To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
>> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>>
>>
>>
>> Ok that does seem like a TLS/SSL issue...
>>
>>
>>
>> Is this a single cluster doing site-to-site to itself?
>>
>>
>>
>> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>>
>> thanks conrad - did get it. Bryan is being more helpful that I so I
>> went silent :-)
>>
>> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>>
>> <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
>> > Hi Joe,
>> > Yep,
>> > Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
>> > That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
>> > Thanks
>> > Conrad
>> >
>> > (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>> >
>> > On 19/10/2016, 14:12, "Joe Witt" <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > Conrad,
>> >
>> > For s2s now you can just point at any of the nodes in the cluster.
>> > Have you tried changing the URL or removing and adding new RPG
>> > entries?
>> >
>> > Thanks
>> > Joe
>> >
>> > On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
>> > <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
>> > > Hi,
>> > >
>> > > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
>> > >
>> > > 6 nodes with a NCM.
>> > >
>> > > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
>> > > my Remote Process Groups work as they previously did because they were
>> > > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
>> > >
>> > > I have tried converting my NCM to a node but whilst I can get it running
>> > > (sort of) when I try and connect to the cluster I get something like this in
>> > > my logs…
>> > >
>> > >
>> > >
>> > > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
>> > > Failed to load flow from cluster due to:
>> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
>> > > node to cluster because local flow is different than cluster flow.
>> > >
>> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
>> > > node to cluster because local flow is different than cluster flow.
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> > > [nifi-jetty-1.0.0.jar:1.0.0]
>> > >
>> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> > > [nifi-runtime-1.0.0.jar:1.0.0]
>> > >
>> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> > > [nifi-runtime-1.0.0.jar:1.0.0]
>> > >
>> > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
>> > > Authorizer is not inheritable by the flow controller because of Authorizer
>> > > differences: Proposed Authorizations do not match current Authorizations
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > ... 4 common frames omitted
>> > >
>> > > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
>> > > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
>> > > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
>> > > to connect node to cluster because local flow is different than cluster
>> > > flow.
>> > >
>> > > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
>> > > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
>> > > Cluster Coordinator,5,main]: java.lang.NullPointerException
>> > >
>> > > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
>> > > org.apache.nifi.NiFi
>> > >
>> > > java.lang.NullPointerException: null
>> > >
>> > > at
>> > > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
>> > > ~[na:1.8.0_51]
>> > >
>> > > at
>> > > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
>> > > ~[na:1.8.0_51]
>> > >
>> > > at
>> > > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
>> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
>> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
>> > >
>> > > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
>> > > Failed to close Leader Selector for Cluster Coordinator
>> > >
>> > > java.lang.IllegalStateException: Already closed or has not been started
>> > >
>> > > at
>> > > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
>> > > ~[guava-18.0.jar:na]
>> > >
>> > > at
>> > > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
>> > > ~[curator-recipes-2.11.0.jar:na]
>> > >
>> > > at
>> > > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
>> > > [nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
>> > > [nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
>> > > [nifi-jetty-1.0.0.jar:1.0.0]
>> > >
>> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> > > [nifi-runtime-1.0.0.jar:1.0.0]
>> > >
>> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> > > [nifi-runtime-1.0.0.jar:1.0.0]
>> > >
>> > > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
>> > > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
>> > > Host due to java.net.SocketException: Socket closed
>> > >
>> > > java.net.SocketException: Socket closed
>> > >
>> > > at java.net.PlainSocketImpl.socketAccept(Native Method)
>> > > ~[na:1.8.0_51]
>> > >
>> > > at
>> > > java.net <http://java.net/>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
>> > > ~[na:1.8.0_51]
>> > >
>> > > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>> > > ~[na:1.8.0_51]
>> > >
>> > > at
>> > > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
>> > > ~[na:1.8.0_51]
>> > >
>> > > at
>> > > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
>> > > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>> > >
>> > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
>> > >
>> > > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
>> > > Failed to start web server... shutting down.
>> > >
>> > > java.lang.Exception: Unable to load flow due to: java.io.IOException:
>> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
>> > > node to cluster because local flow is different than cluster flow.
>> > >
>> > > at
>> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
>> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> > >
>> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
>> > > [nifi-runtime-1.0.0.jar:1.0.0]
>> > >
>> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
>> > > [nifi-runtime-1.0.0.jar:1.0.0]
>> > >
>> > > Caused by: java.io.IOException:
>> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
>> > > node to cluster because local flow is different than cluster flow.
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
>> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
>> > >
>> > > ... 2 common frames omitted
>> > >
>> > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
>> > > connect node to cluster because local flow is different than cluster flow.
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > ... 3 common frames omitted
>> > >
>> > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
>> > > Authorizer is not inheritable by the flow controller because of Authorizer
>> > > differences: Proposed Authorizations do not match current Authorizations
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > at
>> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
>> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
>> > >
>> > > ... 4 common frames omitted
>> > >
>> > > [root@ncm-cm1 logs]#
>> > >
>> > >
>> > >
>> > > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
>> > > part of the problem too.
>> > >
>> > > The docs weren’t very clear on whether (when upgrading and using the legacy
>> > > support of the authorized-user.xml path required the nodes to be also added
>> > > to the authorizers.xml.
>> > >
>> > > I did add them in the end as various attempts to get the cluster up and
>> > > running without them failed (as each server didn’t seem to have rights to do
>> > > anything.
>> > >
>> > >
>> > >
>> > > I have a lot of RPG in my work flows as I am ingesting many syslog data
>> > > sources and this was the recommended pattern to distribute the data
>> > > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
>> > > flow),
>> > >
>> > >
>> > >
>> > > Any suggestions on where to start trying to get this working?
>> > >
>> > > I’ve tried creating a new RPG on one on the datanodes and connecting the
>> > > syslog to that which sort of worked but then I have a bunch of other errors
>> > > when trying to enable the ports to do with not being able to connect to
>> > > (what was) the NCM.
>> > >
>> > >
>> > >
>> > > Thanks
>> > >
>> > > Conrad
>> > >
>> > >
>> > >
>> > > SecureData, combating cyber threats
>> > >
>> > > ________________________________
>> > >
>> > > The information contained in this message or any of its attachments may be
>> > > privileged and confidential and intended for the exclusive use of the
>> > > intended recipient. If you are not the intended recipient any disclosure,
>> > > reproduction, distribution or other dissemination or use of this
>> > > communications is strictly prohibited. The views expressed in this email are
>> > > those of the individual and not necessarily of SecureData Europe Ltd. Any
>> > > prices quoted are only valid if followed up by a formal written quote.
>> > >
>> > > SecureData Europe Limited. Registered in England & Wales 04365896.
>> > > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
>> > > Maidstone, Kent, ME16 9NT
>> >
>> >
>> > ***This email originated outside SecureData***
>> >
>> > Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A== <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
>> >
>> >
>> >
>> >
>>
>>
>>
>>
>>
>>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Andy LoPresto <al...@apache.org>.
Hi Conrad,
Bryan is correct that changing the certificates (and the encapsulating keystores and truststores) will not affect any data held in the nodes.
Regenerating everything using the TLS toolkit should hopefully not be too challenging, but I am also curious as to why you are getting these handshake exceptions now. As Bryan pointed out, adding the following line to bootstrap.conf will provide substantial additional log output which should help trace the issue.
java.arg.15=-Djavax.net.debug=ssl,handshake
You can also imitate the node connecting to the (previous) NCM via this command:
$ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
Where:
<host:port> = the hostname and port of the “NCM”
<path_to_your_cert.pem> = the public key used to identify the “node” (can be exported from the node keystore [1])
<path_to_your_key.pem> = the private key used to identify the “node” (can be exported from the node keystore via 2 step process)
<path_to_your_CA_cert.pem> = the public key used to sign the “NCM” certificate (could be a 3rd party like Verisign or DigiCert, or an internal organization CA if you have one)
If you’ve already regenerated everything and it works, that’s fine. But if you have the time to try and investigate the old certs, we are interested and prepared to help. Thanks.
[1] https://security.stackexchange.com/a/66865/16485
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Oct 19, 2016, at 1:03 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> That is definitely weird that it is only an issue on the node that used to be the NCM.
> Might be worth double checking the keystore and truststore of that one node and make sure it has what you would expect, and also double check nifi.properties compared to the others to see if anything seems different.
>
> Changing all of the keystores, truststores, etc should be fine from a data perspective...
>
> If you decide to go that route it would probably be easiest to start back over from a security perspective, meaning:
> - Stop all the nodes and delete the users.xml and authorizations.xml from all nodes
> - Configure authorizers.xml with the appropriate initial admin (or legacy file) and node identities based on the new certs
> - Ensure authorizers.xml is the same on all nodes
> - Then restart everything
>
> Alternatively, you might be able to manually add users for all of the new certs before shutting everything down and give them the appropriate policies, then restart everything, but requires more manual work to get everything lined up.
>
>
> On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
> Hi,
>
> As a plan for tomorrow – I have generated new keystores, truststores, client certts etc. for all nodes in my cluster using the
>
>
>
> From: Bryan Bende <bbende@gmail.com <ma...@gmail.com>>
> Reply-To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Date: Wednesday, 19 October 2016 at 15:33
>
>
> To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Trying to think of things to check here...
>
>
>
> Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL?
>
>
>
> On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
>
> One other thing…
>
> The RPGs have an unlocked padlock on them saying S2S is not secure.
>
> Conrad
>
>
>
> From: Bryan Bende <bbende@gmail.com <ma...@gmail.com>>
> Reply-To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Date: Wednesday, 19 October 2016 at 15:20
> To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
> Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Ok that does seem like a TLS/SSL issue...
>
>
>
> Is this a single cluster doing site-to-site to itself?
>
>
>
> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>
> thanks conrad - did get it. Bryan is being more helpful that I so I
> went silent :-)
>
> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>
> <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
> > Hi Joe,
> > Yep,
> > Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
> > That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> > Thanks
> > Conrad
> >
> > (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
> >
> > On 19/10/2016, 14:12, "Joe Witt" <joe.witt@gmail.com <ma...@gmail.com>> wrote:
> >
> > Conrad,
> >
> > For s2s now you can just point at any of the nodes in the cluster.
> > Have you tried changing the URL or removing and adding new RPG
> > entries?
> >
> > Thanks
> > Joe
> >
> > On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> > <conrad.crampton@secdata.com <ma...@secdata.com>> wrote:
> > > Hi,
> > >
> > > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> > >
> > > 6 nodes with a NCM.
> > >
> > > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > > my Remote Process Groups work as they previously did because they were
> > > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> > >
> > > I have tried converting my NCM to a node but whilst I can get it running
> > > (sort of) when I try and connect to the cluster I get something like this in
> > > my logs…
> > >
> > >
> > >
> > > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > > Failed to load flow from cluster due to:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > > node to cluster because local flow is different than cluster flow.
> > >
> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > > node to cluster because local flow is different than cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > > Authorizer is not inheritable by the flow controller because of Authorizer
> > > differences: Proposed Authorizations do not match current Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> > > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > > to connect node to cluster because local flow is different than cluster
> > > flow.
> > >
> > > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > > Cluster Coordinator,5,main]: java.lang.NullPointerException
> > >
> > > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi
> > >
> > > java.lang.NullPointerException: null
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > > Failed to close Leader Selector for Cluster Coordinator
> > >
> > > java.lang.IllegalStateException: Already closed or has not been started
> > >
> > > at
> > > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > > ~[guava-18.0.jar:na]
> > >
> > > at
> > > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > > ~[curator-recipes-2.11.0.jar:na]
> > >
> > > at
> > > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > > Host due to java.net.SocketException: Socket closed
> > >
> > > java.net.SocketException: Socket closed
> > >
> > > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.net <http://java.net/>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > > ~[na:1.8.0_51]
> > >
> > > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > > Failed to start web server... shutting down.
> > >
> > > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > > node to cluster because local flow is different than cluster flow.
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > > node to cluster because local flow is different than cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > ... 2 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > > connect node to cluster because local flow is different than cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 3 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > > Authorizer is not inheritable by the flow controller because of Authorizer
> > > differences: Proposed Authorizations do not match current Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > [root@ncm-cm1 logs]#
> > >
> > >
> > >
> > > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > > part of the problem too.
> > >
> > > The docs weren’t very clear on whether (when upgrading and using the legacy
> > > support of the authorized-user.xml path required the nodes to be also added
> > > to the authorizers.xml.
> > >
> > > I did add them in the end as various attempts to get the cluster up and
> > > running without them failed (as each server didn’t seem to have rights to do
> > > anything.
> > >
> > >
> > >
> > > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > > sources and this was the recommended pattern to distribute the data
> > > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > > flow),
> > >
> > >
> > >
> > > Any suggestions on where to start trying to get this working?
> > >
> > > I’ve tried creating a new RPG on one on the datanodes and connecting the
> > > syslog to that which sort of worked but then I have a bunch of other errors
> > > when trying to enable the ports to do with not being able to connect to
> > > (what was) the NCM.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Conrad
> > >
> > >
> > >
> > > SecureData, combating cyber threats
> > >
> > > ________________________________
> > >
> > > The information contained in this message or any of its attachments may be
> > > privileged and confidential and intended for the exclusive use of the
> > > intended recipient. If you are not the intended recipient any disclosure,
> > > reproduction, distribution or other dissemination or use of this
> > > communications is strictly prohibited. The views expressed in this email are
> > > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > > prices quoted are only valid if followed up by a formal written quote.
> > >
> > > SecureData Europe Limited. Registered in England & Wales 04365896.
> > > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > > Maidstone, Kent, ME16 9NT
> >
> >
> > ***This email originated outside SecureData***
> >
> > Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A== <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
> >
> >
> >
> >
>
>
>
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Bryan Bende <bb...@gmail.com>.
That is definitely weird that it is only an issue on the node that used to
be the NCM.
Might be worth double checking the keystore and truststore of that one node
and make sure it has what you would expect, and also double check
nifi.properties compared to the others to see if anything seems different.
Changing all of the keystores, truststores, etc should be fine from a data
perspective...
If you decide to go that route it would probably be easiest to start back
over from a security perspective, meaning:
- Stop all the nodes and delete the users.xml and authorizations.xml from
all nodes
- Configure authorizers.xml with the appropriate initial admin (or legacy
file) and node identities based on the new certs
- Ensure authorizers.xml is the same on all nodes
- Then restart everything
Alternatively, you might be able to manually add users for all of the new
certs before shutting everything down and give them the appropriate
policies, then restart everything, but requires more manual work to get
everything lined up.
On Wed, Oct 19, 2016 at 11:52 AM, Conrad Crampton <
conrad.crampton@secdata.com> wrote:
> Hi,
>
> As a plan for tomorrow – I have generated new keystores, truststores,
> client certts etc. for all nodes in my cluster using the
>
>
>
> *From: *Bryan Bende <bb...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Wednesday, 19 October 2016 at 15:33
>
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Trying to think of things to check here...
>
>
>
> Does every node have nifi.remote.input.secure=true in nifi.properties and
> the URL in the RPG is an https URL?
>
>
>
> On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <
> conrad.crampton@secdata.com> wrote:
>
> One other thing…
>
> The RPGs have an unlocked padlock on them saying S2S is not secure.
>
> Conrad
>
>
>
> *From: *Bryan Bende <bb...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Wednesday, 19 October 2016 at 15:20
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Ok that does seem like a TLS/SSL issue...
>
>
>
> Is this a single cluster doing site-to-site to itself?
>
>
>
> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com> wrote:
>
> thanks conrad - did get it. Bryan is being more helpful that I so I
> went silent :-)
>
> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>
> <co...@secdata.com> wrote:
> > Hi Joe,
> > Yep,
> > Tried removing the RPG that referenced the NCM and adding new one
> with one of the datanodes as url.
> > That sort of worked, but kept getting errors about the NCM not being
> available for the ports and therefore couldn’t actually enable the port I
> needed to for that RPG.
> > Thanks
> > Conrad
> >
> > (sending again as don’t know if the stupid header ‘spoofed’ is stopping
> getting though – apologies if already sent)
> >
> > On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
> >
> > Conrad,
> >
> > For s2s now you can just point at any of the nodes in the
> cluster.
> > Have you tried changing the URL or removing and adding new RPG
> > entries?
> >
> > Thanks
> > Joe
> >
> > On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> > <co...@secdata.com> wrote:
> > > Hi,
> > >
> > > I have finally taken the plunge to upgrade my cluster from
> 0.6.1 to 1.0.0.
> > >
> > > 6 nodes with a NCM.
> > >
> > > With the removal of NCM in 1.0.0 I believe I now have an issue
> where none of
> > > my Remote Process Groups work as they previously did because
> they were
> > > configured to connect to the NCM (as the RPG url) which now
> doesn’t exist.
> > >
> > > I have tried converting my NCM to a node but whilst I can get
> it running
> > > (sort of) when I try and connect to the cluster I get
> something like this in
> > > my logs…
> > >
> > >
> > >
> > > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.
> StandardFlowService
> > > Failed to load flow from cluster due to:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:746)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > > Authorizer is not inheritable by the flow controller because
> of Authorizer
> > > differences: Proposed Authorizations do not match current
> Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDA
> O.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.
> NodeClusterCoordinator
> > > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node
> disconnected from
> > > cluster due to org.apache.nifi.controller.UninheritableFlowException:
> Failed
> > > to connect node to cluster because local flow is different
> than cluster
> > > flow.
> > >
> > > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
> Thread[Shutdown
> > > Cluster Coordinator,5,main]: java.lang.NullPointerException
> > >
> > > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi
> > >
> > > java.lang.NullPointerException: null
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.putVal(
> ConcurrentHashMap.java:1011)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.put(
> ConcurrentHashMap.java:1006)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.
> NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.
> NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService$1.run(
> StandardFlowService.java:330)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> ~[na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.
> CuratorLeaderElectionManager
> > > Failed to close Leader Selector for Cluster Coordinator
> > >
> > > java.lang.IllegalStateException: Already closed or has not
> been started
> > >
> > > at
> > > com.google.common.base.Preconditions.checkState(
> Preconditions.java:173)
> > > ~[guava-18.0.jar:na]
> > >
> > > at
> > > org.apache.curator.framework.recipes.leader.LeaderSelector.
> close(LeaderSelector.java:270)
> > > ~[curator-recipes-2.11.0.jar:na]
> > >
> > > at
> > > org.apache.nifi.controller.leader.election.
> CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.shutdown(
> FlowController.java:1303)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.stop(
> StandardFlowService.java:339)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:753)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > > org.apache.nifi.io.socket.SocketListener Failed to
> communicate with Unknown
> > > Host due to java.net.SocketException: Socket closed
> > >
> > > java.net.SocketException: Socket closed
> > >
> > > at java.net.PlainSocketImpl.socketAccept(Native
> Method)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.net.AbstractPlainSocketImpl.accept(
> AbstractPlainSocketImpl.java:404)
> > > ~[na:1.8.0_51]
> > >
> > > at java.net.ServerSocket.implAccept(ServerSocket.java:
> 545)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > sun.security.ssl.SSLServerSocketImpl.accept(
> SSLServerSocketImpl.java:348)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.io.socket.SocketListener$2.run(
> SocketListener.java:112)
> > > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.
> JettyServer
> > > Failed to start web server... shutting down.
> > >
> > > java.lang.Exception: Unable to load flow due to:
> java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:755)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:497)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:746)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > ... 2 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Failed to
> > > connect node to cluster because local flow is different than
> cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 3 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > > Authorizer is not inheritable by the flow controller because
> of Authorizer
> > > differences: Proposed Authorizations do not match current
> Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDA
> O.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > [root@ncm-cm1 logs]#
> > >
> > >
> > >
> > > I don’t know if the ‘Proposed Authorizer is not inheritable…’
> exception is
> > > part of the problem too.
> > >
> > > The docs weren’t very clear on whether (when upgrading and
> using the legacy
> > > support of the authorized-user.xml path required the nodes to
> be also added
> > > to the authorizers.xml.
> > >
> > > I did add them in the end as various attempts to get the
> cluster up and
> > > running without them failed (as each server didn’t seem to
> have rights to do
> > > anything.
> > >
> > >
> > >
> > > I have a lot of RPG in my work flows as I am ingesting many
> syslog data
> > > sources and this was the recommended pattern to distribute the
> data
> > > (listensyslog…run on primary, output to port (RPG), pick up in
> rest of data
> > > flow),
> > >
> > >
> > >
> > > Any suggestions on where to start trying to get this working?
> > >
> > > I’ve tried creating a new RPG on one on the datanodes and
> connecting the
> > > syslog to that which sort of worked but then I have a bunch of
> other errors
> > > when trying to enable the ports to do with not being able to
> connect to
> > > (what was) the NCM.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Conrad
> > >
> > >
> > >
> > > SecureData, combating cyber threats
> > >
> > > ________________________________
> > >
> > > The information contained in this message or any of its
> attachments may be
> > > privileged and confidential and intended for the exclusive use
> of the
> > > intended recipient. If you are not the intended recipient any
> disclosure,
> > > reproduction, distribution or other dissemination or use of
> this
> > > communications is strictly prohibited. The views expressed in
> this email are
> > > those of the individual and not necessarily of SecureData
> Europe Ltd. Any
> > > prices quoted are only valid if followed up by a formal
> written quote.
> > >
> > > SecureData Europe Limited. Registered in England & Wales
> 04365896.
> > > Registered Address: SecureData House, Hermitage Court,
> Hermitage Lane,
> > > Maidstone, Kent, ME16 9NT
> >
> >
> > ***This email originated outside SecureData***
> >
> > Click https://www.mailcontrol.com/sr/tAj77!!
> uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25L
> WvxYjTGeVGm43FW9a3A==
> <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this
> email as spam.
> >
> >
> >
> >
>
>
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Hi,
As a plan for tomorrow – I have generated new keystores, truststores, client certts etc. for all nodes in my cluster using the
From: Bryan Bende <bb...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 19 October 2016 at 15:33
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Trying to think of things to check here...
Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL?
On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <co...@secdata.com>> wrote:
One other thing…
The RPGs have an unlocked padlock on them saying S2S is not secure.
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:20
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Ok that does seem like a TLS/SSL issue...
Is this a single cluster doing site-to-site to itself?
On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>> wrote:
thanks conrad - did get it. Bryan is being more helpful that I so I
went silent :-)
On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
<co...@secdata.com>> wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
> That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com>> wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com>> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net<http://java.net>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > connect node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the legacy
> > support of the authorized-user.xml path required the nodes to be also added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the datanodes and connecting the
> > syslog to that which sort of worked but then I have a bunch of other errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
>
> ***This email originated outside SecureData***
>
> Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
>
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Hi,
The nifi-user.log doesn’t show any errors – in fact it shows success for any authentication to the old NCM server. What is odd though is the old NCM server is the only one out of the 7 servers that I can’t log into at
https://xxxx:9090/nifi where I can with all the others on their respective ports and hostnames.
I’ll give SSL debug a go, but as a plan for tomorrow – I have generated new keystores, truststores, client certts etc. for all nodes in my cluster using the nifi-toolkit.
Would it be worth using all these newly created ones or will it break existing flowfiles and data held in queues etc.?
Thanks for your help so far.
Conrad
From: Bryan Bende <bb...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 19 October 2016 at 16:38
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Yes http site-to-site was added recently so setting that to disabled should be fine and not related.
If you are using all the same keystores and truststores from before, then I can't think of why the nodes wouldn't be able to communicate securely.
Unless anyone else has some other ideas, you may need to turn on SSL debug (-Djavax.net.debug=all) to see why the handshake is failing.
Is there anything interesting/related in nifi-user.log?
On Wed, Oct 19, 2016 at 10:38 AM, Conrad Crampton <co...@secdata.com>> wrote:
Hi,
Yes, every nifi.properties is set thus – with host and port different for each.
# Site to Site properties
nifi.remote.input.socket.host=ncm.xxxxxxx
nifi.remote.input.socket.port=9870
nifi.remote.input.secure=true
nifi.remote.input.http.enabled=false
nifi.remote.input.http.transaction.ttl=30 sec
You’ll obviously notice that I have http disabled. I set this as this was a new setting which I didn’t have before (it was only RAW in previous versions wasn’t it?)
Does this make a difference?
Thanks
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:33
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Trying to think of things to check here...
Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL?
On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <co...@secdata.com>> wrote:
One other thing…
The RPGs have an unlocked padlock on them saying S2S is not secure.
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:20
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Ok that does seem like a TLS/SSL issue...
Is this a single cluster doing site-to-site to itself?
On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>> wrote:
thanks conrad - did get it. Bryan is being more helpful that I so I
went silent :-)
On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
<co...@secdata.com>> wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
> That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com>> wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com>> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net<http://java.net>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > connect node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the legacy
> > support of the authorized-user.xml path required the nodes to be also added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the datanodes and connecting the
> > syslog to that which sort of worked but then I have a bunch of other errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
>
> ***This email originated outside SecureData***
>
> Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
>
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Bryan Bende <bb...@gmail.com>.
Yes http site-to-site was added recently so setting that to disabled should
be fine and not related.
If you are using all the same keystores and truststores from before, then I
can't think of why the nodes wouldn't be able to communicate securely.
Unless anyone else has some other ideas, you may need to turn on SSL
debug (-Djavax.net.debug=all) to see why the handshake is failing.
Is there anything interesting/related in nifi-user.log?
On Wed, Oct 19, 2016 at 10:38 AM, Conrad Crampton <
conrad.crampton@secdata.com> wrote:
> Hi,
>
> Yes, every nifi.properties is set thus – with host and port different for
> each.
>
>
>
> # Site to Site properties
>
> nifi.remote.input.socket.host=ncm.xxxxxxx
>
> nifi.remote.input.socket.port=9870
>
> nifi.remote.input.secure=true
>
> nifi.remote.input.http.enabled=false
>
> nifi.remote.input.http.transaction.ttl=30 sec
>
>
>
> You’ll obviously notice that I have http disabled. I set this as this was
> a new setting which I didn’t have before (it was only RAW in previous
> versions wasn’t it?)
>
>
>
> Does this make a difference?
>
>
>
> Thanks
>
> Conrad
>
>
>
> *From: *Bryan Bende <bb...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Wednesday, 19 October 2016 at 15:33
>
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Trying to think of things to check here...
>
>
>
> Does every node have nifi.remote.input.secure=true in nifi.properties and
> the URL in the RPG is an https URL?
>
>
>
> On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <
> conrad.crampton@secdata.com> wrote:
>
> One other thing…
>
> The RPGs have an unlocked padlock on them saying S2S is not secure.
>
> Conrad
>
>
>
> *From: *Bryan Bende <bb...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Wednesday, 19 October 2016 at 15:20
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Ok that does seem like a TLS/SSL issue...
>
>
>
> Is this a single cluster doing site-to-site to itself?
>
>
>
> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com> wrote:
>
> thanks conrad - did get it. Bryan is being more helpful that I so I
> went silent :-)
>
> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>
> <co...@secdata.com> wrote:
> > Hi Joe,
> > Yep,
> > Tried removing the RPG that referenced the NCM and adding new one
> with one of the datanodes as url.
> > That sort of worked, but kept getting errors about the NCM not being
> available for the ports and therefore couldn’t actually enable the port I
> needed to for that RPG.
> > Thanks
> > Conrad
> >
> > (sending again as don’t know if the stupid header ‘spoofed’ is stopping
> getting though – apologies if already sent)
> >
> > On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
> >
> > Conrad,
> >
> > For s2s now you can just point at any of the nodes in the
> cluster.
> > Have you tried changing the URL or removing and adding new RPG
> > entries?
> >
> > Thanks
> > Joe
> >
> > On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> > <co...@secdata.com> wrote:
> > > Hi,
> > >
> > > I have finally taken the plunge to upgrade my cluster from
> 0.6.1 to 1.0.0.
> > >
> > > 6 nodes with a NCM.
> > >
> > > With the removal of NCM in 1.0.0 I believe I now have an issue
> where none of
> > > my Remote Process Groups work as they previously did because
> they were
> > > configured to connect to the NCM (as the RPG url) which now
> doesn’t exist.
> > >
> > > I have tried converting my NCM to a node but whilst I can get
> it running
> > > (sort of) when I try and connect to the cluster I get
> something like this in
> > > my logs…
> > >
> > >
> > >
> > > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.
> StandardFlowService
> > > Failed to load flow from cluster due to:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:746)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > > Authorizer is not inheritable by the flow controller because
> of Authorizer
> > > differences: Proposed Authorizations do not match current
> Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDA
> O.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.
> NodeClusterCoordinator
> > > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node
> disconnected from
> > > cluster due to org.apache.nifi.controller.UninheritableFlowException:
> Failed
> > > to connect node to cluster because local flow is different
> than cluster
> > > flow.
> > >
> > > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
> Thread[Shutdown
> > > Cluster Coordinator,5,main]: java.lang.NullPointerException
> > >
> > > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi
> > >
> > > java.lang.NullPointerException: null
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.putVal(
> ConcurrentHashMap.java:1011)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.put(
> ConcurrentHashMap.java:1006)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.
> NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.
> NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService$1.run(
> StandardFlowService.java:330)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> ~[na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.
> CuratorLeaderElectionManager
> > > Failed to close Leader Selector for Cluster Coordinator
> > >
> > > java.lang.IllegalStateException: Already closed or has not
> been started
> > >
> > > at
> > > com.google.common.base.Preconditions.checkState(
> Preconditions.java:173)
> > > ~[guava-18.0.jar:na]
> > >
> > > at
> > > org.apache.curator.framework.recipes.leader.LeaderSelector.
> close(LeaderSelector.java:270)
> > > ~[curator-recipes-2.11.0.jar:na]
> > >
> > > at
> > > org.apache.nifi.controller.leader.election.
> CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.shutdown(
> FlowController.java:1303)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.stop(
> StandardFlowService.java:339)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:753)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > > org.apache.nifi.io.socket.SocketListener Failed to
> communicate with Unknown
> > > Host due to java.net.SocketException: Socket closed
> > >
> > > java.net.SocketException: Socket closed
> > >
> > > at java.net.PlainSocketImpl.socketAccept(Native
> Method)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.net.AbstractPlainSocketImpl.accept(
> AbstractPlainSocketImpl.java:404)
> > > ~[na:1.8.0_51]
> > >
> > > at java.net.ServerSocket.implAccept(ServerSocket.java:
> 545)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > sun.security.ssl.SSLServerSocketImpl.accept(
> SSLServerSocketImpl.java:348)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.io.socket.SocketListener$2.run(
> SocketListener.java:112)
> > > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.
> JettyServer
> > > Failed to start web server... shutting down.
> > >
> > > java.lang.Exception: Unable to load flow due to:
> java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:755)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:497)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:746)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > ... 2 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Failed to
> > > connect node to cluster because local flow is different than
> cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 3 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > > Authorizer is not inheritable by the flow controller because
> of Authorizer
> > > differences: Proposed Authorizations do not match current
> Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDA
> O.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > [root@ncm-cm1 logs]#
> > >
> > >
> > >
> > > I don’t know if the ‘Proposed Authorizer is not inheritable…’
> exception is
> > > part of the problem too.
> > >
> > > The docs weren’t very clear on whether (when upgrading and
> using the legacy
> > > support of the authorized-user.xml path required the nodes to
> be also added
> > > to the authorizers.xml.
> > >
> > > I did add them in the end as various attempts to get the
> cluster up and
> > > running without them failed (as each server didn’t seem to
> have rights to do
> > > anything.
> > >
> > >
> > >
> > > I have a lot of RPG in my work flows as I am ingesting many
> syslog data
> > > sources and this was the recommended pattern to distribute the
> data
> > > (listensyslog…run on primary, output to port (RPG), pick up in
> rest of data
> > > flow),
> > >
> > >
> > >
> > > Any suggestions on where to start trying to get this working?
> > >
> > > I’ve tried creating a new RPG on one on the datanodes and
> connecting the
> > > syslog to that which sort of worked but then I have a bunch of
> other errors
> > > when trying to enable the ports to do with not being able to
> connect to
> > > (what was) the NCM.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Conrad
> > >
> > >
> > >
> > > SecureData, combating cyber threats
> > >
> > > ________________________________
> > >
> > > The information contained in this message or any of its
> attachments may be
> > > privileged and confidential and intended for the exclusive use
> of the
> > > intended recipient. If you are not the intended recipient any
> disclosure,
> > > reproduction, distribution or other dissemination or use of
> this
> > > communications is strictly prohibited. The views expressed in
> this email are
> > > those of the individual and not necessarily of SecureData
> Europe Ltd. Any
> > > prices quoted are only valid if followed up by a formal
> written quote.
> > >
> > > SecureData Europe Limited. Registered in England & Wales
> 04365896.
> > > Registered Address: SecureData House, Hermitage Court,
> Hermitage Lane,
> > > Maidstone, Kent, ME16 9NT
> >
> >
> > ***This email originated outside SecureData***
> >
> > Click https://www.mailcontrol.com/sr/tAj77!!
> uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25L
> WvxYjTGeVGm43FW9a3A==
> <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this
> email as spam.
> >
> >
> >
> >
>
>
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Hi,
Yes, every nifi.properties is set thus – with host and port different for each.
# Site to Site properties
nifi.remote.input.socket.host=ncm.xxxxxxx
nifi.remote.input.socket.port=9870
nifi.remote.input.secure=true
nifi.remote.input.http.enabled=false
nifi.remote.input.http.transaction.ttl=30 sec
You’ll obviously notice that I have http disabled. I set this as this was a new setting which I didn’t have before (it was only RAW in previous versions wasn’t it?)
Does this make a difference?
Thanks
Conrad
From: Bryan Bende <bb...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 19 October 2016 at 15:33
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Trying to think of things to check here...
Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL?
On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <co...@secdata.com>> wrote:
One other thing…
The RPGs have an unlocked padlock on them saying S2S is not secure.
Conrad
From: Bryan Bende <bb...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, 19 October 2016 at 15:20
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Ok that does seem like a TLS/SSL issue...
Is this a single cluster doing site-to-site to itself?
On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>> wrote:
thanks conrad - did get it. Bryan is being more helpful that I so I
went silent :-)
On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
<co...@secdata.com>> wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
> That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com>> wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com>> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net<http://java.net>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > connect node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the legacy
> > support of the authorized-user.xml path required the nodes to be also added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the datanodes and connecting the
> > syslog to that which sort of worked but then I have a bunch of other errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
>
> ***This email originated outside SecureData***
>
> Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
>
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Bryan Bende <bb...@gmail.com>.
Trying to think of things to check here...
Does every node have nifi.remote.input.secure=true in nifi.properties and
the URL in the RPG is an https URL?
On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton <
conrad.crampton@secdata.com> wrote:
> One other thing…
>
> The RPGs have an unlocked padlock on them saying S2S is not secure.
>
> Conrad
>
>
>
> *From: *Bryan Bende <bb...@gmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Wednesday, 19 October 2016 at 15:20
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
>
>
>
> Ok that does seem like a TLS/SSL issue...
>
>
>
> Is this a single cluster doing site-to-site to itself?
>
>
>
> On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com> wrote:
>
> thanks conrad - did get it. Bryan is being more helpful that I so I
> went silent :-)
>
> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
>
> <co...@secdata.com> wrote:
> > Hi Joe,
> > Yep,
> > Tried removing the RPG that referenced the NCM and adding new one
> with one of the datanodes as url.
> > That sort of worked, but kept getting errors about the NCM not being
> available for the ports and therefore couldn’t actually enable the port I
> needed to for that RPG.
> > Thanks
> > Conrad
> >
> > (sending again as don’t know if the stupid header ‘spoofed’ is stopping
> getting though – apologies if already sent)
> >
> > On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
> >
> > Conrad,
> >
> > For s2s now you can just point at any of the nodes in the
> cluster.
> > Have you tried changing the URL or removing and adding new RPG
> > entries?
> >
> > Thanks
> > Joe
> >
> > On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> > <co...@secdata.com> wrote:
> > > Hi,
> > >
> > > I have finally taken the plunge to upgrade my cluster from
> 0.6.1 to 1.0.0.
> > >
> > > 6 nodes with a NCM.
> > >
> > > With the removal of NCM in 1.0.0 I believe I now have an issue
> where none of
> > > my Remote Process Groups work as they previously did because
> they were
> > > configured to connect to the NCM (as the RPG url) which now
> doesn’t exist.
> > >
> > > I have tried converting my NCM to a node but whilst I can get
> it running
> > > (sort of) when I try and connect to the cluster I get
> something like this in
> > > my logs…
> > >
> > >
> > >
> > > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.
> StandardFlowService
> > > Failed to load flow from cluster due to:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:746)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > > Authorizer is not inheritable by the flow controller because
> of Authorizer
> > > differences: Proposed Authorizations do not match current
> Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDA
> O.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.
> NodeClusterCoordinator
> > > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node
> disconnected from
> > > cluster due to org.apache.nifi.controller.UninheritableFlowException:
> Failed
> > > to connect node to cluster because local flow is different
> than cluster
> > > flow.
> > >
> > > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
> Thread[Shutdown
> > > Cluster Coordinator,5,main]: java.lang.NullPointerException
> > >
> > > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi
> > >
> > > java.lang.NullPointerException: null
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.putVal(
> ConcurrentHashMap.java:1011)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.put(
> ConcurrentHashMap.java:1006)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.
> NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.
> NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService$1.run(
> StandardFlowService.java:330)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> ~[na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.
> CuratorLeaderElectionManager
> > > Failed to close Leader Selector for Cluster Coordinator
> > >
> > > java.lang.IllegalStateException: Already closed or has not
> been started
> > >
> > > at
> > > com.google.common.base.Preconditions.checkState(
> Preconditions.java:173)
> > > ~[guava-18.0.jar:na]
> > >
> > > at
> > > org.apache.curator.framework.recipes.leader.LeaderSelector.
> close(LeaderSelector.java:270)
> > > ~[curator-recipes-2.11.0.jar:na]
> > >
> > > at
> > > org.apache.nifi.controller.leader.election.
> CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.shutdown(
> FlowController.java:1303)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.stop(
> StandardFlowService.java:339)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:753)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > > org.apache.nifi.io.socket.SocketListener Failed to
> communicate with Unknown
> > > Host due to java.net.SocketException: Socket closed
> > >
> > > java.net.SocketException: Socket closed
> > >
> > > at java.net.PlainSocketImpl.socketAccept(Native
> Method)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.net.AbstractPlainSocketImpl.accept(
> AbstractPlainSocketImpl.java:404)
> > > ~[na:1.8.0_51]
> > >
> > > at java.net.ServerSocket.implAccept(ServerSocket.java:
> 545)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > sun.security.ssl.SSLServerSocketImpl.accept(
> SSLServerSocketImpl.java:348)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.io.socket.SocketListener$2.run(
> SocketListener.java:112)
> > > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.
> JettyServer
> > > Failed to start web server... shutting down.
> > >
> > > java.lang.Exception: Unable to load flow due to:
> java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:755)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:497)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:746)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > ... 2 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Failed to
> > > connect node to cluster because local flow is different than
> cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 3 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > > Authorizer is not inheritable by the flow controller because
> of Authorizer
> > > differences: Proposed Authorizations do not match current
> Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDA
> O.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > [root@ncm-cm1 logs]#
> > >
> > >
> > >
> > > I don’t know if the ‘Proposed Authorizer is not inheritable…’
> exception is
> > > part of the problem too.
> > >
> > > The docs weren’t very clear on whether (when upgrading and
> using the legacy
> > > support of the authorized-user.xml path required the nodes to
> be also added
> > > to the authorizers.xml.
> > >
> > > I did add them in the end as various attempts to get the
> cluster up and
> > > running without them failed (as each server didn’t seem to
> have rights to do
> > > anything.
> > >
> > >
> > >
> > > I have a lot of RPG in my work flows as I am ingesting many
> syslog data
> > > sources and this was the recommended pattern to distribute the
> data
> > > (listensyslog…run on primary, output to port (RPG), pick up in
> rest of data
> > > flow),
> > >
> > >
> > >
> > > Any suggestions on where to start trying to get this working?
> > >
> > > I’ve tried creating a new RPG on one on the datanodes and
> connecting the
> > > syslog to that which sort of worked but then I have a bunch of
> other errors
> > > when trying to enable the ports to do with not being able to
> connect to
> > > (what was) the NCM.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Conrad
> > >
> > >
> > >
> > > SecureData, combating cyber threats
> > >
> > > ________________________________
> > >
> > > The information contained in this message or any of its
> attachments may be
> > > privileged and confidential and intended for the exclusive use
> of the
> > > intended recipient. If you are not the intended recipient any
> disclosure,
> > > reproduction, distribution or other dissemination or use of
> this
> > > communications is strictly prohibited. The views expressed in
> this email are
> > > those of the individual and not necessarily of SecureData
> Europe Ltd. Any
> > > prices quoted are only valid if followed up by a formal
> written quote.
> > >
> > > SecureData Europe Limited. Registered in England & Wales
> 04365896.
> > > Registered Address: SecureData House, Hermitage Court,
> Hermitage Lane,
> > > Maidstone, Kent, ME16 9NT
> >
> >
> > ***This email originated outside SecureData***
> >
> > Click https://www.mailcontrol.com/sr/tAj77!!
> uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25L
> WvxYjTGeVGm43FW9a3A==
> <https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this
> email as spam.
> >
> >
> >
> >
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
One other thing…
The RPGs have an unlocked padlock on them saying S2S is not secure.
Conrad
From: Bryan Bende <bb...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 19 October 2016 at 15:20
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Ok that does seem like a TLS/SSL issue...
Is this a single cluster doing site-to-site to itself?
On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>> wrote:
thanks conrad - did get it. Bryan is being more helpful that I so I
went silent :-)
On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
<co...@secdata.com>> wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
> That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com>> wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com>> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net<http://java.net>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > connect node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the legacy
> > support of the authorized-user.xml path required the nodes to be also added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the datanodes and connecting the
> > syslog to that which sort of worked but then I have a bunch of other errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
>
> ***This email originated outside SecureData***
>
> Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
>
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Yes.
All 6 nodes and 1 NCM were the original set up
After upgrade not fussed whether I lost the NCM and went to just the 6 nodes or re-introduced the redundant NCM as a new node (for actually doing stuff). I had to go the latter route in the end as the RPG were complaining about it not being there.
So the S2S stuff and RPG are only there to split the data after being received by primary node.
Thanks
Conrad
From: Bryan Bende <bb...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Wednesday, 19 October 2016 at 15:20
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Ok that does seem like a TLS/SSL issue...
Is this a single cluster doing site-to-site to itself?
On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com>> wrote:
thanks conrad - did get it. Bryan is being more helpful that I so I
went silent :-)
On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
<co...@secdata.com>> wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
> That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com>> wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com>> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net<http://java.net>.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > connect node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the legacy
> > support of the authorized-user.xml path required the nodes to be also added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the datanodes and connecting the
> > syslog to that which sort of worked but then I have a bunch of other errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
>
> ***This email originated outside SecureData***
>
> Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A==<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
>
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Bryan Bende <bb...@gmail.com>.
Ok that does seem like a TLS/SSL issue...
Is this a single cluster doing site-to-site to itself?
On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt <jo...@gmail.com> wrote:
> thanks conrad - did get it. Bryan is being more helpful that I so I
> went silent :-)
>
> On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
> <co...@secdata.com> wrote:
> > Hi Joe,
> > Yep,
> > Tried removing the RPG that referenced the NCM and adding new one
> with one of the datanodes as url.
> > That sort of worked, but kept getting errors about the NCM not being
> available for the ports and therefore couldn’t actually enable the port I
> needed to for that RPG.
> > Thanks
> > Conrad
> >
> > (sending again as don’t know if the stupid header ‘spoofed’ is stopping
> getting though – apologies if already sent)
> >
> > On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
> >
> > Conrad,
> >
> > For s2s now you can just point at any of the nodes in the
> cluster.
> > Have you tried changing the URL or removing and adding new RPG
> > entries?
> >
> > Thanks
> > Joe
> >
> > On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> > <co...@secdata.com> wrote:
> > > Hi,
> > >
> > > I have finally taken the plunge to upgrade my cluster from
> 0.6.1 to 1.0.0.
> > >
> > > 6 nodes with a NCM.
> > >
> > > With the removal of NCM in 1.0.0 I believe I now have an issue
> where none of
> > > my Remote Process Groups work as they previously did because
> they were
> > > configured to connect to the NCM (as the RPG url) which now
> doesn’t exist.
> > >
> > > I have tried converting my NCM to a node but whilst I can get
> it running
> > > (sort of) when I try and connect to the cluster I get
> something like this in
> > > my logs…
> > >
> > >
> > >
> > > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.
> StandardFlowService
> > > Failed to load flow from cluster due to:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:746)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > > Authorizer is not inheritable by the flow controller because
> of Authorizer
> > > differences: Proposed Authorizations do not match current
> Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDA
> O.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.
> NodeClusterCoordinator
> > > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node
> disconnected from
> > > cluster due to org.apache.nifi.controller.UninheritableFlowException:
> Failed
> > > to connect node to cluster because local flow is different
> than cluster
> > > flow.
> > >
> > > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi An Unknown Error Occurred in Thread
> Thread[Shutdown
> > > Cluster Coordinator,5,main]: java.lang.NullPointerException
> > >
> > > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > > org.apache.nifi.NiFi
> > >
> > > java.lang.NullPointerException: null
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.putVal(
> ConcurrentHashMap.java:1011)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.util.concurrent.ConcurrentHashMap.put(
> ConcurrentHashMap.java:1006)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.
> NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.cluster.coordination.node.
> NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService$1.run(
> StandardFlowService.java:330)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> ~[na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.
> CuratorLeaderElectionManager
> > > Failed to close Leader Selector for Cluster Coordinator
> > >
> > > java.lang.IllegalStateException: Already closed or has not
> been started
> > >
> > > at
> > > com.google.common.base.Preconditions.checkState(
> Preconditions.java:173)
> > > ~[guava-18.0.jar:na]
> > >
> > > at
> > > org.apache.curator.framework.recipes.leader.LeaderSelector.
> close(LeaderSelector.java:270)
> > > ~[curator-recipes-2.11.0.jar:na]
> > >
> > > at
> > > org.apache.nifi.controller.leader.election.
> CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.shutdown(
> FlowController.java:1303)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.stop(
> StandardFlowService.java:339)
> > > [nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:753)
> > > [nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > > org.apache.nifi.io.socket.SocketListener Failed to
> communicate with Unknown
> > > Host due to java.net.SocketException: Socket closed
> > >
> > > java.net.SocketException: Socket closed
> > >
> > > at java.net.PlainSocketImpl.socketAccept(Native
> Method)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > java.net.AbstractPlainSocketImpl.accept(
> AbstractPlainSocketImpl.java:404)
> > > ~[na:1.8.0_51]
> > >
> > > at java.net.ServerSocket.implAccept(ServerSocket.java:
> 545)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > sun.security.ssl.SSLServerSocketImpl.accept(
> SSLServerSocketImpl.java:348)
> > > ~[na:1.8.0_51]
> > >
> > > at
> > > org.apache.nifi.io.socket.SocketListener$2.run(
> SocketListener.java:112)
> > > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> > >
> > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> > >
> > > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.
> JettyServer
> > > Failed to start web server... shutting down.
> > >
> > > java.lang.Exception: Unable to load flow due to:
> java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:755)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > > [nifi-runtime-1.0.0.jar:1.0.0]
> > >
> > > Caused by: java.io.IOException:
> > > org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect
> > > node to cluster because local flow is different than cluster
> flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:497)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.web.server.JettyServer.start(JettyServer.
> java:746)
> > > ~[nifi-jetty-1.0.0.jar:1.0.0]
> > >
> > > ... 2 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Failed to
> > > connect node to cluster because local flow is different than
> cluster flow.
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:879)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.load(
> StandardFlowService.java:493)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 3 common frames omitted
> > >
> > > Caused by: org.apache.nifi.controller.UninheritableFlowException:
> Proposed
> > > Authorizer is not inheritable by the flow controller because
> of Authorizer
> > > differences: Proposed Authorizations do not match current
> Authorizations
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowSynchronizer.sync(
> StandardFlowSynchronizer.java:252)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.FlowController.synchronize(
> FlowController.java:1435)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.persistence.StandardXMLFlowConfigurationDA
> O.load(StandardXMLFlowConfigurationDAO.java:83)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.loadFromBytes(
> StandardFlowService.java:671)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > at
> > > org.apache.nifi.controller.StandardFlowService.
> loadFromConnectionResponse(StandardFlowService.java:857)
> > > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> > >
> > > ... 4 common frames omitted
> > >
> > > [root@ncm-cm1 logs]#
> > >
> > >
> > >
> > > I don’t know if the ‘Proposed Authorizer is not inheritable…’
> exception is
> > > part of the problem too.
> > >
> > > The docs weren’t very clear on whether (when upgrading and
> using the legacy
> > > support of the authorized-user.xml path required the nodes to
> be also added
> > > to the authorizers.xml.
> > >
> > > I did add them in the end as various attempts to get the
> cluster up and
> > > running without them failed (as each server didn’t seem to
> have rights to do
> > > anything.
> > >
> > >
> > >
> > > I have a lot of RPG in my work flows as I am ingesting many
> syslog data
> > > sources and this was the recommended pattern to distribute the
> data
> > > (listensyslog…run on primary, output to port (RPG), pick up in
> rest of data
> > > flow),
> > >
> > >
> > >
> > > Any suggestions on where to start trying to get this working?
> > >
> > > I’ve tried creating a new RPG on one on the datanodes and
> connecting the
> > > syslog to that which sort of worked but then I have a bunch of
> other errors
> > > when trying to enable the ports to do with not being able to
> connect to
> > > (what was) the NCM.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Conrad
> > >
> > >
> > >
> > > SecureData, combating cyber threats
> > >
> > > ________________________________
> > >
> > > The information contained in this message or any of its
> attachments may be
> > > privileged and confidential and intended for the exclusive use
> of the
> > > intended recipient. If you are not the intended recipient any
> disclosure,
> > > reproduction, distribution or other dissemination or use of
> this
> > > communications is strictly prohibited. The views expressed in
> this email are
> > > those of the individual and not necessarily of SecureData
> Europe Ltd. Any
> > > prices quoted are only valid if followed up by a formal
> written quote.
> > >
> > > SecureData Europe Limited. Registered in England & Wales
> 04365896.
> > > Registered Address: SecureData House, Hermitage Court,
> Hermitage Lane,
> > > Maidstone, Kent, ME16 9NT
> >
> >
> > ***This email originated outside SecureData***
> >
> > Click https://www.mailcontrol.com/sr/tAj77!!
> uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25L
> WvxYjTGeVGm43FW9a3A== to report this email as spam.
> >
> >
> >
> >
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Joe Witt <jo...@gmail.com>.
thanks conrad - did get it. Bryan is being more helpful that I so I
went silent :-)
On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton
<co...@secdata.com> wrote:
> Hi Joe,
> Yep,
> Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
> That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
> Thanks
> Conrad
>
> (sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
>
> On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
>
> Conrad,
>
> For s2s now you can just point at any of the nodes in the cluster.
> Have you tried changing the URL or removing and adding new RPG
> entries?
>
> Thanks
> Joe
>
> On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
> <co...@secdata.com> wrote:
> > Hi,
> >
> > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
> >
> > 6 nodes with a NCM.
> >
> > With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> > my Remote Process Groups work as they previously did because they were
> > configured to connect to the NCM (as the RPG url) which now doesn’t exist.
> >
> > I have tried converting my NCM to a node but whilst I can get it running
> > (sort of) when I try and connect to the cluster I get something like this in
> > my logs…
> >
> >
> >
> > 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> > Failed to load flow from cluster due to:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> > Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> > cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> > to connect node to cluster because local flow is different than cluster
> > flow.
> >
> > 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> > Cluster Coordinator,5,main]: java.lang.NullPointerException
> >
> > 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> > org.apache.nifi.NiFi
> >
> > java.lang.NullPointerException: null
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> > ~[na:1.8.0_51]
> >
> > at
> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> > ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
> >
> > 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> > Failed to close Leader Selector for Cluster Coordinator
> >
> > java.lang.IllegalStateException: Already closed or has not been started
> >
> > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> > ~[guava-18.0.jar:na]
> >
> > at
> > org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> > ~[curator-recipes-2.11.0.jar:na]
> >
> > at
> > org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> > [nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> > [nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> > org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> > Host due to java.net.SocketException: Socket closed
> >
> > java.net.SocketException: Socket closed
> >
> > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > ~[na:1.8.0_51]
> >
> > at
> > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> > ~[na:1.8.0_51]
> >
> > at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> > ~[na:1.8.0_51]
> >
> > at
> > sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> > ~[na:1.8.0_51]
> >
> > at
> > org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> >
> > 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> > Failed to start web server... shutting down.
> >
> > java.lang.Exception: Unable to load flow due to: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > at org.apache.nifi.NiFi.main(NiFi.java:243)
> > [nifi-runtime-1.0.0.jar:1.0.0]
> >
> > Caused by: java.io.IOException:
> > org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> > node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> > ~[nifi-jetty-1.0.0.jar:1.0.0]
> >
> > ... 2 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> > connect node to cluster because local flow is different than cluster flow.
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 3 common frames omitted
> >
> > Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> > Authorizer is not inheritable by the flow controller because of Authorizer
> > differences: Proposed Authorizations do not match current Authorizations
> >
> > at
> > org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > at
> > org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> > ~[nifi-framework-core-1.0.0.jar:1.0.0]
> >
> > ... 4 common frames omitted
> >
> > [root@ncm-cm1 logs]#
> >
> >
> >
> > I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> > part of the problem too.
> >
> > The docs weren’t very clear on whether (when upgrading and using the legacy
> > support of the authorized-user.xml path required the nodes to be also added
> > to the authorizers.xml.
> >
> > I did add them in the end as various attempts to get the cluster up and
> > running without them failed (as each server didn’t seem to have rights to do
> > anything.
> >
> >
> >
> > I have a lot of RPG in my work flows as I am ingesting many syslog data
> > sources and this was the recommended pattern to distribute the data
> > (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> > flow),
> >
> >
> >
> > Any suggestions on where to start trying to get this working?
> >
> > I’ve tried creating a new RPG on one on the datanodes and connecting the
> > syslog to that which sort of worked but then I have a bunch of other errors
> > when trying to enable the ports to do with not being able to connect to
> > (what was) the NCM.
> >
> >
> >
> > Thanks
> >
> > Conrad
> >
> >
> >
> > SecureData, combating cyber threats
> >
> > ________________________________
> >
> > The information contained in this message or any of its attachments may be
> > privileged and confidential and intended for the exclusive use of the
> > intended recipient. If you are not the intended recipient any disclosure,
> > reproduction, distribution or other dissemination or use of this
> > communications is strictly prohibited. The views expressed in this email are
> > those of the individual and not necessarily of SecureData Europe Ltd. Any
> > prices quoted are only valid if followed up by a formal written quote.
> >
> > SecureData Europe Limited. Registered in England & Wales 04365896.
> > Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> > Maidstone, Kent, ME16 9NT
>
>
> ***This email originated outside SecureData***
>
> Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A== to report this email as spam.
>
>
>
>
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Hi Joe,
Yep,
Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
Thanks
Conrad
(sending again as don’t know if the stupid header ‘spoofed’ is stopping getting though – apologies if already sent)
On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
Conrad,
For s2s now you can just point at any of the nodes in the cluster.
Have you tried changing the URL or removing and adding new RPG
entries?
Thanks
Joe
On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
<co...@secdata.com> wrote:
> Hi,
>
> I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
>
> 6 nodes with a NCM.
>
> With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> my Remote Process Groups work as they previously did because they were
> configured to connect to the NCM (as the RPG url) which now doesn’t exist.
>
> I have tried converting my NCM to a node but whilst I can get it running
> (sort of) when I try and connect to the cluster I get something like this in
> my logs…
>
>
>
> 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> Failed to load flow from cluster due to:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> [nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> Authorizer is not inheritable by the flow controller because of Authorizer
> differences: Proposed Authorizations do not match current Authorizations
>
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 4 common frames omitted
>
> 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect node to cluster because local flow is different than cluster
> flow.
>
> 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> Cluster Coordinator,5,main]: java.lang.NullPointerException
>
> 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi
>
> java.lang.NullPointerException: null
>
> at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> ~[na:1.8.0_51]
>
> at
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> ~[na:1.8.0_51]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
>
> 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> Failed to close Leader Selector for Cluster Coordinator
>
> java.lang.IllegalStateException: Already closed or has not been started
>
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> ~[guava-18.0.jar:na]
>
> at
> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> ~[curator-recipes-2.11.0.jar:na]
>
> at
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> [nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> Host due to java.net.SocketException: Socket closed
>
> java.net.SocketException: Socket closed
>
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> ~[na:1.8.0_51]
>
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> ~[na:1.8.0_51]
>
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> ~[na:1.8.0_51]
>
> at
> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> ~[na:1.8.0_51]
>
> at
> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
>
> 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
>
> java.lang.Exception: Unable to load flow due to: java.io.IOException:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> ~[nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> Caused by: java.io.IOException:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> ~[nifi-jetty-1.0.0.jar:1.0.0]
>
> ... 2 common frames omitted
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> connect node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 3 common frames omitted
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> Authorizer is not inheritable by the flow controller because of Authorizer
> differences: Proposed Authorizations do not match current Authorizations
>
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 4 common frames omitted
>
> [root@ncm-cm1 logs]#
>
>
>
> I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> part of the problem too.
>
> The docs weren’t very clear on whether (when upgrading and using the legacy
> support of the authorized-user.xml path required the nodes to be also added
> to the authorizers.xml.
>
> I did add them in the end as various attempts to get the cluster up and
> running without them failed (as each server didn’t seem to have rights to do
> anything.
>
>
>
> I have a lot of RPG in my work flows as I am ingesting many syslog data
> sources and this was the recommended pattern to distribute the data
> (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> flow),
>
>
>
> Any suggestions on where to start trying to get this working?
>
> I’ve tried creating a new RPG on one on the datanodes and connecting the
> syslog to that which sort of worked but then I have a bunch of other errors
> when trying to enable the ports to do with not being able to connect to
> (what was) the NCM.
>
>
>
> Thanks
>
> Conrad
>
>
>
> SecureData, combating cyber threats
>
> ________________________________
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email are
> those of the individual and not necessarily of SecureData Europe Ltd. Any
> prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT
***This email originated outside SecureData***
Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A== to report this email as spam.
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Conrad Crampton <co...@SecData.com>.
Hi Joe,
Yep,
Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url.
That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.
Thanks
Conrad
On 19/10/2016, 14:12, "Joe Witt" <jo...@gmail.com> wrote:
Conrad,
For s2s now you can just point at any of the nodes in the cluster.
Have you tried changing the URL or removing and adding new RPG
entries?
Thanks
Joe
On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
<co...@secdata.com> wrote:
> Hi,
>
> I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
>
> 6 nodes with a NCM.
>
> With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> my Remote Process Groups work as they previously did because they were
> configured to connect to the NCM (as the RPG url) which now doesn’t exist.
>
> I have tried converting my NCM to a node but whilst I can get it running
> (sort of) when I try and connect to the cluster I get something like this in
> my logs…
>
>
>
> 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> Failed to load flow from cluster due to:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> [nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> Authorizer is not inheritable by the flow controller because of Authorizer
> differences: Proposed Authorizations do not match current Authorizations
>
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 4 common frames omitted
>
> 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect node to cluster because local flow is different than cluster
> flow.
>
> 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> Cluster Coordinator,5,main]: java.lang.NullPointerException
>
> 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi
>
> java.lang.NullPointerException: null
>
> at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> ~[na:1.8.0_51]
>
> at
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> ~[na:1.8.0_51]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
>
> 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> Failed to close Leader Selector for Cluster Coordinator
>
> java.lang.IllegalStateException: Already closed or has not been started
>
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> ~[guava-18.0.jar:na]
>
> at
> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> ~[curator-recipes-2.11.0.jar:na]
>
> at
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> [nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> Host due to java.net.SocketException: Socket closed
>
> java.net.SocketException: Socket closed
>
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> ~[na:1.8.0_51]
>
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> ~[na:1.8.0_51]
>
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> ~[na:1.8.0_51]
>
> at
> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> ~[na:1.8.0_51]
>
> at
> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
>
> 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
>
> java.lang.Exception: Unable to load flow due to: java.io.IOException:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> ~[nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> Caused by: java.io.IOException:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> ~[nifi-jetty-1.0.0.jar:1.0.0]
>
> ... 2 common frames omitted
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> connect node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 3 common frames omitted
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> Authorizer is not inheritable by the flow controller because of Authorizer
> differences: Proposed Authorizations do not match current Authorizations
>
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 4 common frames omitted
>
> [root@ncm-cm1 logs]#
>
>
>
> I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> part of the problem too.
>
> The docs weren’t very clear on whether (when upgrading and using the legacy
> support of the authorized-user.xml path required the nodes to be also added
> to the authorizers.xml.
>
> I did add them in the end as various attempts to get the cluster up and
> running without them failed (as each server didn’t seem to have rights to do
> anything.
>
>
>
> I have a lot of RPG in my work flows as I am ingesting many syslog data
> sources and this was the recommended pattern to distribute the data
> (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> flow),
>
>
>
> Any suggestions on where to start trying to get this working?
>
> I’ve tried creating a new RPG on one on the datanodes and connecting the
> syslog to that which sort of worked but then I have a bunch of other errors
> when trying to enable the ports to do with not being able to connect to
> (what was) the NCM.
>
>
>
> Thanks
>
> Conrad
>
>
>
> SecureData, combating cyber threats
>
> ________________________________
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email are
> those of the individual and not necessarily of SecureData Europe Ltd. Any
> prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT
***This email originated outside SecureData***
Click https://www.mailcontrol.com/sr/tAj77!!uP0XGX2PQPOmvUu5zZAYN1Mos55ZMH65vS49VoLnJlQAkvDtaSciXa9lO25LWvxYjTGeVGm43FW9a3A== to report this email as spam.
Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups
Posted by Joe Witt <jo...@gmail.com>.
Conrad,
For s2s now you can just point at any of the nodes in the cluster.
Have you tried changing the URL or removing and adding new RPG
entries?
Thanks
Joe
On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton
<co...@secdata.com> wrote:
> Hi,
>
> I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0.
>
> 6 nodes with a NCM.
>
> With the removal of NCM in 1.0.0 I believe I now have an issue where none of
> my Remote Process Groups work as they previously did because they were
> configured to connect to the NCM (as the RPG url) which now doesn’t exist.
>
> I have tried converting my NCM to a node but whilst I can get it running
> (sort of) when I try and connect to the cluster I get something like this in
> my logs…
>
>
>
> 2016-10-19 13:14:44,109 ERROR [main] o.a.nifi.controller.StandardFlowService
> Failed to load flow from cluster due to:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> [nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> Authorizer is not inheritable by the flow controller because of Authorizer
> differences: Proposed Authorizations do not match current Authorizations
>
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 4 common frames omitted
>
> 2016-10-19 13:14:44,414 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator
> Event Reported for ncm-cm1.mis-cds.local:9090 -- Node disconnected from
> cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed
> to connect node to cluster because local flow is different than cluster
> flow.
>
> 2016-10-19 13:14:44,420 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Shutdown
> Cluster Coordinator,5,main]: java.lang.NullPointerException
>
> 2016-10-19 13:14:44,423 ERROR [Shutdown Cluster Coordinator]
> org.apache.nifi.NiFi
>
> java.lang.NullPointerException: null
>
> at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> ~[na:1.8.0_51]
>
> at
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> ~[na:1.8.0_51]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.updateNodeStatus(NodeClusterCoordinator.java:570)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator.shutdown(NodeClusterCoordinator.java:119)
> ~[nifi-framework-cluster-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService$1.run(StandardFlowService.java:330)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
>
> 2016-10-19 13:14:44,448 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager
> Failed to close Leader Selector for Cluster Coordinator
>
> java.lang.IllegalStateException: Already closed or has not been started
>
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:173)
> ~[guava-18.0.jar:na]
>
> at
> org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
> ~[curator-recipes-2.11.0.jar:na]
>
> at
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:159)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1303)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:339)
> [nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:753)
> [nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> 2016-10-19 13:14:45,062 WARN [Cluster Socket Listener]
> org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown
> Host due to java.net.SocketException: Socket closed
>
> java.net.SocketException: Socket closed
>
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> ~[na:1.8.0_51]
>
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
> ~[na:1.8.0_51]
>
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> ~[na:1.8.0_51]
>
> at
> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
> ~[na:1.8.0_51]
>
> at
> org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
>
> 2016-10-19 13:14:45,064 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
>
> java.lang.Exception: Unable to load flow due to: java.io.IOException:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:755)
> ~[nifi-jetty-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.<init>(NiFi.java:152)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> [nifi-runtime-1.0.0.jar:1.0.0]
>
> Caused by: java.io.IOException:
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect
> node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:497)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746)
> ~[nifi-jetty-1.0.0.jar:1.0.0]
>
> ... 2 common frames omitted
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Failed to
> connect node to cluster because local flow is different than cluster flow.
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:879)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 3 common frames omitted
>
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed
> Authorizer is not inheritable by the flow controller because of Authorizer
> differences: Proposed Authorizations do not match current Authorizations
>
> at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:252)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1435)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:671)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:857)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
>
> ... 4 common frames omitted
>
> [root@ncm-cm1 logs]#
>
>
>
> I don’t know if the ‘Proposed Authorizer is not inheritable…’ exception is
> part of the problem too.
>
> The docs weren’t very clear on whether (when upgrading and using the legacy
> support of the authorized-user.xml path required the nodes to be also added
> to the authorizers.xml.
>
> I did add them in the end as various attempts to get the cluster up and
> running without them failed (as each server didn’t seem to have rights to do
> anything.
>
>
>
> I have a lot of RPG in my work flows as I am ingesting many syslog data
> sources and this was the recommended pattern to distribute the data
> (listensyslog…run on primary, output to port (RPG), pick up in rest of data
> flow),
>
>
>
> Any suggestions on where to start trying to get this working?
>
> I’ve tried creating a new RPG on one on the datanodes and connecting the
> syslog to that which sort of worked but then I have a bunch of other errors
> when trying to enable the ports to do with not being able to connect to
> (what was) the NCM.
>
>
>
> Thanks
>
> Conrad
>
>
>
> SecureData, combating cyber threats
>
> ________________________________
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email are
> those of the individual and not necessarily of SecureData Europe Ltd. Any
> prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT