You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by rammohan ganapavarapu <ra...@gmail.com> on 2020/01/21 21:50:18 UTC
Zookeeper resolving to old host IP addresses
Hi,
Does zookeeper cache the host IP? if so how long does it cache? I have a
zk cluster in autoscaling groups and when a new node comes up, other nodes
still resolving to old IP. Is there any setting to invalidate dns cache for
zookeeper? or is it jvm dns cache? until other nodes resolves to new IP,
this node not able to join the cluster.
Thanks,
Ram
Re: Zookeeper resolving to old host IP addresses
Posted by rammohan ganapavarapu <ra...@gmail.com>.
Andor,
Thank you, i will look into dynamic reconfig option.
Ram
On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar <an...@apache.org> wrote:
> Hi Ram,
>
> As far as I can see from the code, ZooKeeper uses the standard Java calls
> getByName() and getAllByName() every time it’s trying to connect to a
> server.
>
> ————————
> // zookeeper.ipReachableTimeout is not defined
> if (ipReachableTimeout <= 0) {
> address = InetAddress.getByName(this.hostname);
> } else {
> address = getReachableAddress(this.hostname, ipReachableTimeout);
> }
> ————————
>
> ZK doesn’t (and definitely should not) cache IP addresses. It’s either the
> cache of JVM or your DNS server.
>
> Dynamic reconfig is available in 3.5.x versions which are already stable
> now and I think with that you don’t need to reuse existing hostnames.
> Instead use reconfig commands to properly remove old nodes and add new
> ones. Sounds like more cumbersome, but maybe more reliable.
>
> Andor
>
>
>
>
> > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
> >
> > But still happening for me, is there any config on zookeeper side to make
> > this fix to work?
> >
> > Ram
> >
> > On Tue, Jan 21, 2020 at 2:12 PM Michael Han <ha...@apache.org> wrote:
> >
> >> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
> >>
> >> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> >> rammohanganap@gmail.com> wrote:
> >>
> >>> Hi Enrico,
> >>>
> >>> I see same with both 3.4.5 and 3.4.14
> >>>
> >>> Ram
> >>>
> >>> On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <eo...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>> Which version of ZK are you using?
> >>>> Enrico
> >>>>
> >>>>
> >>>> Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> >> rammohanganap@gmail.com
> >>>>
> >>>> ha scritto:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> Does zookeeper cache the host IP? if so how long does it cache? I
> >> have
> >>> a
> >>>>> zk cluster in autoscaling groups and when a new node comes up, other
> >>>> nodes
> >>>>> still resolving to old IP. Is there any setting to invalidate dns
> >> cache
> >>>> for
> >>>>> zookeeper? or is it jvm dns cache? until other nodes resolves to new
> >>> IP,
> >>>>> this node not able to join the cluster.
> >>>>>
> >>>>> Thanks,
> >>>>> Ram
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: Zookeeper resolving to old host IP addresses
Posted by Zhewei Hu <zh...@gmail.com>.
Hi Ram,
I also faced this issue before. I think JVM only resolves the DNS records
once when ZK service starts. After that, even if you update the DNS record
to a new IP address and the ZK server itself is able to resolve the new DNS
record, the JVM will not pick up the new IP address. I think the only way is
to restart the ZK service.
You could use IP addresses instead of DNS records in your ZK config. In this
way, each time you want to replace a ZK server, you could launch a new
server with the same IP address. This could fix the issue.
Btw, with the dynamic config feature introduced in ZK 3.5+, ZK should be
able to recognize the IP address of the new ZK server automatically.
HTH,
Zhewei
--
Sent from: http://zookeeper-user.578899.n2.nabble.com/
Re: Zookeeper resolving to old host IP addresses
Posted by rammohan ganapavarapu <ra...@gmail.com>.
Andor,
It didnt work, my OS is resolving to new IP but for some reason zk
processes is still resolving to old IP address even with those two
properties. Any other pointers?
Thanks,
Ram
On Wed, Jan 22, 2020 at 1:18 PM Andor Molnar <an...@apache.org> wrote:
> Yep, give it a try.
>
>
> https://cr.openjdk.java.net/~iris/se/11/latestSpec/api/java.base/java/net/doc-files/net-properties.html
>
> I don't have personal experience with these settings.
>
> Andor
>
>
>
>
> On Wed, 2020-01-22 at 10:37 -0800, rammohan ganapavarapu wrote:
> > Hi Andor,
> >
> > On OS side the hostname resolves to new IP, so it could be JVM is
> > the one
> > caching. Any setting son jvm to invalidate cache? In some other posts
> > i did
> > see some one recommending these but not sure it it works.
> >
> > -Dnetworkaddress.cache.ttl=0
> >
> > -Dnetworkaddress.cache.negative.ttl=0
> >
> > Ram
> >
> > On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar <an...@apache.org>
> > wrote:
> >
> > > Hi Ram,
> > >
> > > As far as I can see from the code, ZooKeeper uses the standard Java
> > > calls
> > > getByName() and getAllByName() every time it’s trying to connect to
> > > a
> > > server.
> > >
> > > ————————
> > > // zookeeper.ipReachableTimeout is not defined
> > > if (ipReachableTimeout <= 0) {
> > > address = InetAddress.getByName(this.hostname);
> > > } else {
> > > address = getReachableAddress(this.hostname,
> > > ipReachableTimeout);
> > > }
> > > ————————
> > >
> > > ZK doesn’t (and definitely should not) cache IP addresses. It’s
> > > either the
> > > cache of JVM or your DNS server.
> > >
> > > Dynamic reconfig is available in 3.5.x versions which are already
> > > stable
> > > now and I think with that you don’t need to reuse existing
> > > hostnames.
> > > Instead use reconfig commands to properly remove old nodes and add
> > > new
> > > ones. Sounds like more cumbersome, but maybe more reliable.
> > >
> > > Andor
> > >
> > >
> > >
> > >
> > > > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> > > rammohanganap@gmail.com> wrote:
> > > > But still happening for me, is there any config on zookeeper side
> > > > to make
> > > > this fix to work?
> > > >
> > > > Ram
> > > >
> > > > On Tue, Jan 21, 2020 at 2:12 PM Michael Han <ha...@apache.org>
> > > > wrote:
> > > >
> > > > > Could be ZOOKEEPER-1506, though this should be fixed already in
> > > > > 3.4.14.
> > > > >
> > > > > On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> > > > > rammohanganap@gmail.com> wrote:
> > > > >
> > > > > > Hi Enrico,
> > > > > >
> > > > > > I see same with both 3.4.5 and 3.4.14
> > > > > >
> > > > > > Ram
> > > > > >
> > > > > > On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <
> > > > > > eolivelli@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > Which version of ZK are you using?
> > > > > > > Enrico
> > > > > > >
> > > > > > >
> > > > > > > Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> > > > > rammohanganap@gmail.com
> > > > > > > ha scritto:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Does zookeeper cache the host IP? if so how long does it
> > > > > > > > cache? I
> > > > > have
> > > > > > a
> > > > > > > > zk cluster in autoscaling groups and when a new node
> > > > > > > > comes up, other
> > > > > > > nodes
> > > > > > > > still resolving to old IP. Is there any setting to
> > > > > > > > invalidate dns
> > > > > cache
> > > > > > > for
> > > > > > > > zookeeper? or is it jvm dns cache? until other nodes
> > > > > > > > resolves to new
> > > > > > IP,
> > > > > > > > this node not able to join the cluster.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Ram
> > > > > > > >
>
>
Re: Zookeeper resolving to old host IP addresses
Posted by Andor Molnar <an...@apache.org>.
Yep, give it a try.
https://cr.openjdk.java.net/~iris/se/11/latestSpec/api/java.base/java/net/doc-files/net-properties.html
I don't have personal experience with these settings.
Andor
On Wed, 2020-01-22 at 10:37 -0800, rammohan ganapavarapu wrote:
> Hi Andor,
>
> On OS side the hostname resolves to new IP, so it could be JVM is
> the one
> caching. Any setting son jvm to invalidate cache? In some other posts
> i did
> see some one recommending these but not sure it it works.
>
> -Dnetworkaddress.cache.ttl=0
>
> -Dnetworkaddress.cache.negative.ttl=0
>
> Ram
>
> On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar <an...@apache.org>
> wrote:
>
> > Hi Ram,
> >
> > As far as I can see from the code, ZooKeeper uses the standard Java
> > calls
> > getByName() and getAllByName() every time it’s trying to connect to
> > a
> > server.
> >
> > ————————
> > // zookeeper.ipReachableTimeout is not defined
> > if (ipReachableTimeout <= 0) {
> > address = InetAddress.getByName(this.hostname);
> > } else {
> > address = getReachableAddress(this.hostname,
> > ipReachableTimeout);
> > }
> > ————————
> >
> > ZK doesn’t (and definitely should not) cache IP addresses. It’s
> > either the
> > cache of JVM or your DNS server.
> >
> > Dynamic reconfig is available in 3.5.x versions which are already
> > stable
> > now and I think with that you don’t need to reuse existing
> > hostnames.
> > Instead use reconfig commands to properly remove old nodes and add
> > new
> > ones. Sounds like more cumbersome, but maybe more reliable.
> >
> > Andor
> >
> >
> >
> >
> > > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> > rammohanganap@gmail.com> wrote:
> > > But still happening for me, is there any config on zookeeper side
> > > to make
> > > this fix to work?
> > >
> > > Ram
> > >
> > > On Tue, Jan 21, 2020 at 2:12 PM Michael Han <ha...@apache.org>
> > > wrote:
> > >
> > > > Could be ZOOKEEPER-1506, though this should be fixed already in
> > > > 3.4.14.
> > > >
> > > > On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> > > > rammohanganap@gmail.com> wrote:
> > > >
> > > > > Hi Enrico,
> > > > >
> > > > > I see same with both 3.4.5 and 3.4.14
> > > > >
> > > > > Ram
> > > > >
> > > > > On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <
> > > > > eolivelli@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > Which version of ZK are you using?
> > > > > > Enrico
> > > > > >
> > > > > >
> > > > > > Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> > > > rammohanganap@gmail.com
> > > > > > ha scritto:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Does zookeeper cache the host IP? if so how long does it
> > > > > > > cache? I
> > > > have
> > > > > a
> > > > > > > zk cluster in autoscaling groups and when a new node
> > > > > > > comes up, other
> > > > > > nodes
> > > > > > > still resolving to old IP. Is there any setting to
> > > > > > > invalidate dns
> > > > cache
> > > > > > for
> > > > > > > zookeeper? or is it jvm dns cache? until other nodes
> > > > > > > resolves to new
> > > > > IP,
> > > > > > > this node not able to join the cluster.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ram
> > > > > > >
Re: Zookeeper resolving to old host IP addresses
Posted by rammohan ganapavarapu <ra...@gmail.com>.
Hi Andor,
On OS side the hostname resolves to new IP, so it could be JVM is the one
caching. Any setting son jvm to invalidate cache? In some other posts i did
see some one recommending these but not sure it it works.
-Dnetworkaddress.cache.ttl=0
-Dnetworkaddress.cache.negative.ttl=0
Ram
On Wed, Jan 22, 2020 at 2:03 AM Andor Molnar <an...@apache.org> wrote:
> Hi Ram,
>
> As far as I can see from the code, ZooKeeper uses the standard Java calls
> getByName() and getAllByName() every time it’s trying to connect to a
> server.
>
> ————————
> // zookeeper.ipReachableTimeout is not defined
> if (ipReachableTimeout <= 0) {
> address = InetAddress.getByName(this.hostname);
> } else {
> address = getReachableAddress(this.hostname, ipReachableTimeout);
> }
> ————————
>
> ZK doesn’t (and definitely should not) cache IP addresses. It’s either the
> cache of JVM or your DNS server.
>
> Dynamic reconfig is available in 3.5.x versions which are already stable
> now and I think with that you don’t need to reuse existing hostnames.
> Instead use reconfig commands to properly remove old nodes and add new
> ones. Sounds like more cumbersome, but maybe more reliable.
>
> Andor
>
>
>
>
> > On 2020. Jan 21., at 23:14, rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
> >
> > But still happening for me, is there any config on zookeeper side to make
> > this fix to work?
> >
> > Ram
> >
> > On Tue, Jan 21, 2020 at 2:12 PM Michael Han <ha...@apache.org> wrote:
> >
> >> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
> >>
> >> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> >> rammohanganap@gmail.com> wrote:
> >>
> >>> Hi Enrico,
> >>>
> >>> I see same with both 3.4.5 and 3.4.14
> >>>
> >>> Ram
> >>>
> >>> On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <eo...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>> Which version of ZK are you using?
> >>>> Enrico
> >>>>
> >>>>
> >>>> Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> >> rammohanganap@gmail.com
> >>>>
> >>>> ha scritto:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> Does zookeeper cache the host IP? if so how long does it cache? I
> >> have
> >>> a
> >>>>> zk cluster in autoscaling groups and when a new node comes up, other
> >>>> nodes
> >>>>> still resolving to old IP. Is there any setting to invalidate dns
> >> cache
> >>>> for
> >>>>> zookeeper? or is it jvm dns cache? until other nodes resolves to new
> >>> IP,
> >>>>> this node not able to join the cluster.
> >>>>>
> >>>>> Thanks,
> >>>>> Ram
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: Zookeeper resolving to old host IP addresses
Posted by Andor Molnar <an...@apache.org>.
Hi Ram,
As far as I can see from the code, ZooKeeper uses the standard Java calls getByName() and getAllByName() every time it’s trying to connect to a server.
————————
// zookeeper.ipReachableTimeout is not defined
if (ipReachableTimeout <= 0) {
address = InetAddress.getByName(this.hostname);
} else {
address = getReachableAddress(this.hostname, ipReachableTimeout);
}
————————
ZK doesn’t (and definitely should not) cache IP addresses. It’s either the cache of JVM or your DNS server.
Dynamic reconfig is available in 3.5.x versions which are already stable now and I think with that you don’t need to reuse existing hostnames. Instead use reconfig commands to properly remove old nodes and add new ones. Sounds like more cumbersome, but maybe more reliable.
Andor
> On 2020. Jan 21., at 23:14, rammohan ganapavarapu <ra...@gmail.com> wrote:
>
> But still happening for me, is there any config on zookeeper side to make
> this fix to work?
>
> Ram
>
> On Tue, Jan 21, 2020 at 2:12 PM Michael Han <ha...@apache.org> wrote:
>
>> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
>>
>> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
>> rammohanganap@gmail.com> wrote:
>>
>>> Hi Enrico,
>>>
>>> I see same with both 3.4.5 and 3.4.14
>>>
>>> Ram
>>>
>>> On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <eo...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> Which version of ZK are you using?
>>>> Enrico
>>>>
>>>>
>>>> Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
>> rammohanganap@gmail.com
>>>>
>>>> ha scritto:
>>>>
>>>>> Hi,
>>>>>
>>>>> Does zookeeper cache the host IP? if so how long does it cache? I
>> have
>>> a
>>>>> zk cluster in autoscaling groups and when a new node comes up, other
>>>> nodes
>>>>> still resolving to old IP. Is there any setting to invalidate dns
>> cache
>>>> for
>>>>> zookeeper? or is it jvm dns cache? until other nodes resolves to new
>>> IP,
>>>>> this node not able to join the cluster.
>>>>>
>>>>> Thanks,
>>>>> Ram
>>>>>
>>>>
>>>
>>
Re: Zookeeper resolving to old host IP addresses
Posted by rammohan ganapavarapu <ra...@gmail.com>.
But still happening for me, is there any config on zookeeper side to make
this fix to work?
Ram
On Tue, Jan 21, 2020 at 2:12 PM Michael Han <ha...@apache.org> wrote:
> Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
>
> On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
>
> > Hi Enrico,
> >
> > I see same with both 3.4.5 and 3.4.14
> >
> > Ram
> >
> > On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > Which version of ZK are you using?
> > > Enrico
> > >
> > >
> > > Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <
> rammohanganap@gmail.com
> > >
> > > ha scritto:
> > >
> > > > Hi,
> > > >
> > > > Does zookeeper cache the host IP? if so how long does it cache? I
> have
> > a
> > > > zk cluster in autoscaling groups and when a new node comes up, other
> > > nodes
> > > > still resolving to old IP. Is there any setting to invalidate dns
> cache
> > > for
> > > > zookeeper? or is it jvm dns cache? until other nodes resolves to new
> > IP,
> > > > this node not able to join the cluster.
> > > >
> > > > Thanks,
> > > > Ram
> > > >
> > >
> >
>
Re: Zookeeper resolving to old host IP addresses
Posted by Michael Han <ha...@apache.org>.
Could be ZOOKEEPER-1506, though this should be fixed already in 3.4.14.
On Tue, Jan 21, 2020 at 2:01 PM rammohan ganapavarapu <
rammohanganap@gmail.com> wrote:
> Hi Enrico,
>
> I see same with both 3.4.5 and 3.4.14
>
> Ram
>
> On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Hi,
> > Which version of ZK are you using?
> > Enrico
> >
> >
> > Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <rammohanganap@gmail.com
> >
> > ha scritto:
> >
> > > Hi,
> > >
> > > Does zookeeper cache the host IP? if so how long does it cache? I have
> a
> > > zk cluster in autoscaling groups and when a new node comes up, other
> > nodes
> > > still resolving to old IP. Is there any setting to invalidate dns cache
> > for
> > > zookeeper? or is it jvm dns cache? until other nodes resolves to new
> IP,
> > > this node not able to join the cluster.
> > >
> > > Thanks,
> > > Ram
> > >
> >
>
Re: Zookeeper resolving to old host IP addresses
Posted by rammohan ganapavarapu <ra...@gmail.com>.
Hi Enrico,
I see same with both 3.4.5 and 3.4.14
Ram
On Tue, Jan 21, 2020 at 1:53 PM Enrico Olivelli <eo...@gmail.com> wrote:
> Hi,
> Which version of ZK are you using?
> Enrico
>
>
> Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <ra...@gmail.com>
> ha scritto:
>
> > Hi,
> >
> > Does zookeeper cache the host IP? if so how long does it cache? I have a
> > zk cluster in autoscaling groups and when a new node comes up, other
> nodes
> > still resolving to old IP. Is there any setting to invalidate dns cache
> for
> > zookeeper? or is it jvm dns cache? until other nodes resolves to new IP,
> > this node not able to join the cluster.
> >
> > Thanks,
> > Ram
> >
>
Re: Zookeeper resolving to old host IP addresses
Posted by Enrico Olivelli <eo...@gmail.com>.
Hi,
Which version of ZK are you using?
Enrico
Il mar 21 gen 2020, 22:51 rammohan ganapavarapu <ra...@gmail.com>
ha scritto:
> Hi,
>
> Does zookeeper cache the host IP? if so how long does it cache? I have a
> zk cluster in autoscaling groups and when a new node comes up, other nodes
> still resolving to old IP. Is there any setting to invalidate dns cache for
> zookeeper? or is it jvm dns cache? until other nodes resolves to new IP,
> this node not able to join the cluster.
>
> Thanks,
> Ram
>