Zookeeper DNS TTL

[Ralph Rodkey <ra...@angieslist.com>]

Is there any way to make Kafka 0.8.2.2 honor the DNS TTLs of its Zookeeper ensemble? I'm specifying the Zookeepers by hostname, but Kafka doesn't seem to ever re-resolve those hostnames after startup. I've tried setting the relevant JVM system properties via "-Dnetworkaddress.cache.ttl=0 -Dnetworkaddress.cache.negative.ttl=0", but it doesn't seem like Kafka's behavior was affected by those options.

I don't see any reference to this issue in the bug tracker. Is there a way to deal with Zookeeper server replacement short of restarting Kafka? Are there other approaches I can take that won't result in downtime? The only option I can think of is to do a rolling restart of the Kafka servers after each Zookeeper server is replaced, which seems distasteful.

Re: Zookeeper DNS TTL

[David DeMaagd <dd...@linkedin.com.INVALID>]

https://issues.apache.org/jira/browse/ZOOKEEPER-1356 which was closed as a
dupe of https://issues.apache.org/jira/browse/ZOOKEEPER-338 are relevant to
this...  It's a zk client issue, and there are things you can do to avoid
having to reconfigure the clients while you're bouncing them (CNAMEs and
the like), but that's not what you're looking for :/

On Fri, Mar 18, 2016 at 10:49 AM, Ralph Rodkey <ra...@angieslist.com>
wrote:

> Is there any way to make Kafka 0.8.2.2 honor the DNS TTLs of its Zookeeper
> ensemble? I'm specifying the Zookeepers by hostname, but Kafka doesn't seem
> to ever re-resolve those hostnames after startup. I've tried setting the
> relevant JVM system properties via "-Dnetworkaddress.cache.ttl=0
> -Dnetworkaddress.cache.negative.ttl=0", but it doesn't seem like Kafka's
> behavior was affected by those options.
>
> I don't see any reference to this issue in the bug tracker. Is there a way
> to deal with Zookeeper server replacement short of restarting Kafka? Are
> there other approaches I can take that won't result in downtime? The only
> option I can think of is to do a rolling restart of the Kafka servers after
> each Zookeeper server is replaced, which seems distasteful.
>



-- 
Dave DeMaagd | S'aite Reliability Engineering, Y'all
ddemaagd@linkedin.com | 818 262 7958

Re: Zookeeper DNS TTL

[Scott Reynolds <sr...@twilio.com>]

Ah I have been down this path. It is the zookeeper client. It resolves and
caches the ip addresses:
https://github.com/apache/zookeeper/blob/bd9a1448f9b29859092e6bdca93da121ec166b7a/src/java/main/org/apache/zookeeper/client/StaticHostProvider.java#L108

I believe they are cached forever.

We have had success with HAProxy to manage connections. We have also
written and patched Kafka to use our service discovery system to always
have an ip to resolve too.

On Fri, Mar 18, 2016 at 10:49 AM Ralph Rodkey <ra...@angieslist.com>
wrote:

> Is there any way to make Kafka 0.8.2.2 honor the DNS TTLs of its Zookeeper
> ensemble? I'm specifying the Zookeepers by hostname, but Kafka doesn't seem
> to ever re-resolve those hostnames after startup. I've tried setting the
> relevant JVM system properties via "-Dnetworkaddress.cache.ttl=0
> -Dnetworkaddress.cache.negative.ttl=0", but it doesn't seem like Kafka's
> behavior was affected by those options.
>
> I don't see any reference to this issue in the bug tracker. Is there a way
> to deal with Zookeeper server replacement short of restarting Kafka? Are
> there other approaches I can take that won't result in downtime? The only
> option I can think of is to do a rolling restart of the Kafka servers after
> each Zookeeper server is replaced, which seems distasteful.
>