host names in a clustered NiFi

["Knapp, Michael" <Mi...@capitalone.com>]

Hi,

I’m trying to run NiFi in a clustered configuration within docker containers on kubernetes.  While I have NiFi starting in standalone mode, I get exceptions when launching it in clustered mode.  I’m having a lot of trouble figuring out what the host names really should be.

I looked through the system properties<http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties> (web and cluster properties) and also the Admin Guide (Cluster setup section).  The description of these host properties is quite vague:

                nifi.cluster.node.address – The fully qualified address of the node.  It is blank by default.
                Nifi.web.http.host – The HTTP host.  It is blank by default.
                Nifi.web.https.host – The HTTPS host. It is blank by default.

The descriptions above are quite vague and don’t explain to me how NiFi will behave based on these values.  In fact, I keep needing to go through a lot of trial and error to make this work.  What happens if they are blank?  What happens if I insert the IP address?

Just asking for the host name is ambiguous, I have a docker internal host name, an EC2 host name, a kubernetes service name, load balancer host names, and a route53 host name.  The correct value really depends on how these are used in the code.  I could easily see these needing to be the kubernetes service name or the internal host name, or the route53 host name.  What should I be using here?  Should all nodes in the cluster have the same value for this?

Here is the stack trace I am seeing at runtime:

2017-08-08 14:23:51,534 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Unresolved address
2017-08-08 14:23:51,535 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut 2017-08-08 14:23:51,536 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut java.net.SocketException: Unresolved address
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.nio.ch.Net.translateToSocketException(Net.java:131)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.nio.ch.Net.translateException(Net.java:157)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.nio.ch.Net.translateException(Net.java:163)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:76)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:298)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.eclipse.jetty.server.Server.doStart(Server.java:431)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:705)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.apache.nifi.NiFi.<init>(NiFi.java:160)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at org.apache.nifi.NiFi.main(NiFi.java:267)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut Caused by: java.nio.channels.UnresolvedAddressException: null
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.nio.ch.Net.checkAddress(Net.java:101)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:218)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
2017-08-08 14:23:51,536 INFO [NiFi logging handler] org.apache.nifi.StdOut ... 9 common frames omitted


Michael Knapp
________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.