Discussion: Address binding for monitor.

[Ed Coleman <de...@etcoleman.com>]

Discovered the way the monitor determines the hostname and publishes address
for the monitor log forwarding, that is written to zookeeper for clients,
changed slightly between 1.6.4 and 1.6.5.

 

In 1.6.4 the monitor uses InetAddress.getLocalHost(). to determine the
hostname that is written to zookeeper for discovery by the tservers.

 

In 1.6.5 it uses the -address command line parameter. This is getting set by
the server-start script, which calls accumulo-env.sh. In accumulo-env.sh we
had the monitor set to bind to all interfaces at the default port, 4560.
This resolved to the monitor binding to 0.0.0.0:4560 - which is correct for
the monitor. However, it then put 0.0.0.0:4560 into zookeeper for the
tservers - at which point they could not publish log messages to the
monitor.

 

Setting the monitor to not bind to all interfaces in accumulo-env.sh, the
server start script then uses the monitor hostname, and this is published
and the loggers had a valid hostname:port to forward logs to. But now, the
monitor is only bound to the interface that resolves to the hostname - if
other interface(s) were used, the monitor is not going to receive log
messages sent to them. (If the interfaces are bound together through the OS,
this is not an issue.)

 

It seems like there should be two parameters to control this behavior - one
for setting the bind address for the monitor, another to set the "external"
address that is published into zookeeper so that tservers can find the
host:port to forward log messages to.

 

It seems that this could be useful for running Accumulo in containers that
may have different "virtual" interface / ports. And there may be other
similar configuration changes that we can consider. Wanted to open a
discussion to see if there are other considerations / requirements and
services that should be considered before any change is recommended.

 

Ed Coleman

 

Re: Discussion: Address binding for monitor.

[Josh Elser <jo...@gmail.com>]

Ed Coleman wrote:
> Discovered the way the monitor determines the hostname and publishes address
> for the monitor log forwarding, that is written to zookeeper for clients,
> changed slightly between 1.6.4 and 1.6.5.

Being (one of?) the people who has been messing with this recently, this 
seems rather unintended.

> In 1.6.4 the monitor uses InetAddress.getLocalHost(). to determine the
> hostname that is written to zookeeper for discovery by the tservers.
>
>
>
> In 1.6.5 it uses the -address command line parameter. This is getting set by
> the server-start script, which calls accumulo-env.sh. In accumulo-env.sh we
> had the monitor set to bind to all interfaces at the default port, 4560.
> This resolved to the monitor binding to 0.0.0.0:4560 - which is correct for
> the monitor. However, it then put 0.0.0.0:4560 into zookeeper for the
> tservers - at which point they could not publish log messages to the
> monitor.

Advertising 0.0.0.0 is just flat out wrong. Determining which interface 
(when presented with multiple) is difficult though..

> Setting the monitor to not bind to all interfaces in accumulo-env.sh, the
> server start script then uses the monitor hostname, and this is published
> and the loggers had a valid hostname:port to forward logs to. But now, the
> monitor is only bound to the interface that resolves to the hostname - if
> other interface(s) were used, the monitor is not going to receive log
> messages sent to them. (If the interfaces are bound together through the OS,
> this is not an issue.)
>
>
>
> It seems like there should be two parameters to control this behavior - one
> for setting the bind address for the monitor, another to set the "external"
> address that is published into zookeeper so that tservers can find the
> host:port to forward log messages to.

At first glance, I would say the quick fix would be to use 
InetAddress.getLocalHost and do the rDNS lookup on that (which, IIRC, 
should use the default name resolution configured on the box and 
/etc/hosts). I'm still not 100% sure how this would work on systems with 
multiple NICs, but I think it would be consistent with how 1.6.4 worked?

> It seems that this could be useful for running Accumulo in containers that
> may have different "virtual" interface / ports. And there may be other
> similar configuration changes that we can consider. Wanted to open a
> discussion to see if there are other considerations / requirements and
> services that should be considered before any change is recommended.

Completely agree that this is something where we don't entirely have the 
necessary configuration in place. It would be great to evaluate what is 
necessary and how best we can improve on what we have now (maybe even 
surveying what other people do).

Thanks for writing this up. This is a really good catch.

> Ed Coleman
>
>
>
>