You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jason Rosenberg <jb...@squareup.com> on 2013/05/21 20:09:28 UTC
Producer can't initialize if zk host down (0.7.2)
Hi,
I am seeing an unexpected situation. My producers use a zkconnection
string to connect to kafka (this is still 0.7.2). If one of the zk hosts
is taken down and removed from dns, it causes an UnknownHostException, and
the producer can't initialize. I expect this is different than the less
severe case where one of the zk's is not responding, but the host still
exists in dns, etc. We are in the process or moving some of our zk
cluster, but I had hoped that it would be ok to do in stages....e.g. as
long as most of the zk's are available in a client's connection string, it
should be ok for it proceed, without having to proactively update and
restart each client with the new zk connect string.
However, this appears not to be the case (see exception below). Is this
expected?
org.I0Itec.zkclient.exception.ZkException: Unable to connect to <redacted>
at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66)
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at
kafka.producer.ZKBrokerPartitionInfo.<init>(ZKBrokerPartitionInfo.scala:62)
at kafka.producer.Producer.<init>(Producer.scala:47)
at kafka.javaapi.producer.Producer.<init>(Producer.scala:33)
at kafka.javaapi.producer.Producer.<init>(Producer.scala:40)
......
Caused by: java.net.UnknownHostException: <redacted>
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850)
at
java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201)
at java.net.InetAddress.getAllByName0(InetAddress.java:1154)
at java.net.InetAddress.getAllByName(InetAddress.java:1084)
at java.net.InetAddress.getAllByName(InetAddress.java:1020)
at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:382)
at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:327)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383)
at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64)
... 19 more
Jason
Re: Producer can't initialize if zk host down (0.7.2)
Posted by Neha Narkhede <ne...@gmail.com>.
Right. Zookeeper has added the feature to allow dynamically reconfiguring
zookeeper clusters, but it is part of zookeeper 3.5.0.
Thanks,
Neha
On Tue, May 21, 2013 at 3:05 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> Thanks Neha,
>
> I see now that is true. It looks like it just needs to resolve all the
> hosts (but doesn't need zk to be available on each host). So
> decommissioning a zknode (and removing from dns) is a bit more sensitive
> than I thought. I can't think of any reason that zkClient should require
> this behavior, other than for keeping clean configurations. But it makes
> it difficult to roll in new changes to the cluster layout while keeping
> services up.
>
> Jason
>
>
> On Tue, May 21, 2013 at 2:20 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > According to zookeeper code, when you try to create a client handle to
> > zookeeper, it resolves each of the hosts in the zookeeper connection
> > string. If any of these fail, it throws an exception. If you use a
> > zookeeper based producer, it tries to create a ZkClient handle which in
> > turn tries to create a ZooKeeper handle. Since the ZooKeeper handle has
> to
> > resolve every host on startup, you have to make sure that is possible.
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, May 21, 2013 at 11:09 AM, Jason Rosenberg <jb...@squareup.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I am seeing an unexpected situation. My producers use a zkconnection
> > > string to connect to kafka (this is still 0.7.2). If one of the zk
> hosts
> > > is taken down and removed from dns, it causes an UnknownHostException,
> > and
> > > the producer can't initialize. I expect this is different than the
> less
> > > severe case where one of the zk's is not responding, but the host still
> > > exists in dns, etc. We are in the process or moving some of our zk
> > > cluster, but I had hoped that it would be ok to do in stages....e.g. as
> > > long as most of the zk's are available in a client's connection string,
> > it
> > > should be ok for it proceed, without having to proactively update and
> > > restart each client with the new zk connect string.
> > >
> > > However, this appears not to be the case (see exception below). Is
> this
> > > expected?
> > >
> > > org.I0Itec.zkclient.exception.ZkException: Unable to connect to
> > <redacted>
> > > at
> org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66)
> > > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872)
> > > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
> > > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
> > > at
> > >
> >
> kafka.producer.ZKBrokerPartitionInfo.<init>(ZKBrokerPartitionInfo.scala:62)
> > > at kafka.producer.Producer.<init>(Producer.scala:47)
> > > at kafka.javaapi.producer.Producer.<init>(Producer.scala:33)
> > > at kafka.javaapi.producer.Producer.<init>(Producer.scala:40)
> > > ......
> > > Caused by: java.net.UnknownHostException: <redacted>
> > > at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> > > at
> java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850)
> > > at
> > > java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201)
> > > at java.net.InetAddress.getAllByName0(InetAddress.java:1154)
> > > at java.net.InetAddress.getAllByName(InetAddress.java:1084)
> > > at java.net.InetAddress.getAllByName(InetAddress.java:1020)
> > > at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:382)
> > > at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:327)
> > > at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383)
> > > at
> org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64)
> > > ... 19 more
> > >
> > > Jason
> > >
> >
>
Re: Producer can't initialize if zk host down (0.7.2)
Posted by Jason Rosenberg <jb...@squareup.com>.
Thanks Neha,
I see now that is true. It looks like it just needs to resolve all the
hosts (but doesn't need zk to be available on each host). So
decommissioning a zknode (and removing from dns) is a bit more sensitive
than I thought. I can't think of any reason that zkClient should require
this behavior, other than for keeping clean configurations. But it makes
it difficult to roll in new changes to the cluster layout while keeping
services up.
Jason
On Tue, May 21, 2013 at 2:20 PM, Neha Narkhede <ne...@gmail.com>wrote:
> According to zookeeper code, when you try to create a client handle to
> zookeeper, it resolves each of the hosts in the zookeeper connection
> string. If any of these fail, it throws an exception. If you use a
> zookeeper based producer, it tries to create a ZkClient handle which in
> turn tries to create a ZooKeeper handle. Since the ZooKeeper handle has to
> resolve every host on startup, you have to make sure that is possible.
>
> Thanks,
> Neha
>
>
> On Tue, May 21, 2013 at 11:09 AM, Jason Rosenberg <jb...@squareup.com>
> wrote:
>
> > Hi,
> >
> > I am seeing an unexpected situation. My producers use a zkconnection
> > string to connect to kafka (this is still 0.7.2). If one of the zk hosts
> > is taken down and removed from dns, it causes an UnknownHostException,
> and
> > the producer can't initialize. I expect this is different than the less
> > severe case where one of the zk's is not responding, but the host still
> > exists in dns, etc. We are in the process or moving some of our zk
> > cluster, but I had hoped that it would be ok to do in stages....e.g. as
> > long as most of the zk's are available in a client's connection string,
> it
> > should be ok for it proceed, without having to proactively update and
> > restart each client with the new zk connect string.
> >
> > However, this appears not to be the case (see exception below). Is this
> > expected?
> >
> > org.I0Itec.zkclient.exception.ZkException: Unable to connect to
> <redacted>
> > at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66)
> > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872)
> > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
> > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
> > at
> >
> kafka.producer.ZKBrokerPartitionInfo.<init>(ZKBrokerPartitionInfo.scala:62)
> > at kafka.producer.Producer.<init>(Producer.scala:47)
> > at kafka.javaapi.producer.Producer.<init>(Producer.scala:33)
> > at kafka.javaapi.producer.Producer.<init>(Producer.scala:40)
> > ......
> > Caused by: java.net.UnknownHostException: <redacted>
> > at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> > at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850)
> > at
> > java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201)
> > at java.net.InetAddress.getAllByName0(InetAddress.java:1154)
> > at java.net.InetAddress.getAllByName(InetAddress.java:1084)
> > at java.net.InetAddress.getAllByName(InetAddress.java:1020)
> > at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:382)
> > at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:327)
> > at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383)
> > at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64)
> > ... 19 more
> >
> > Jason
> >
>
Re: Producer can't initialize if zk host down (0.7.2)
Posted by Neha Narkhede <ne...@gmail.com>.
According to zookeeper code, when you try to create a client handle to
zookeeper, it resolves each of the hosts in the zookeeper connection
string. If any of these fail, it throws an exception. If you use a
zookeeper based producer, it tries to create a ZkClient handle which in
turn tries to create a ZooKeeper handle. Since the ZooKeeper handle has to
resolve every host on startup, you have to make sure that is possible.
Thanks,
Neha
On Tue, May 21, 2013 at 11:09 AM, Jason Rosenberg <jb...@squareup.com> wrote:
> Hi,
>
> I am seeing an unexpected situation. My producers use a zkconnection
> string to connect to kafka (this is still 0.7.2). If one of the zk hosts
> is taken down and removed from dns, it causes an UnknownHostException, and
> the producer can't initialize. I expect this is different than the less
> severe case where one of the zk's is not responding, but the host still
> exists in dns, etc. We are in the process or moving some of our zk
> cluster, but I had hoped that it would be ok to do in stages....e.g. as
> long as most of the zk's are available in a client's connection string, it
> should be ok for it proceed, without having to proactively update and
> restart each client with the new zk connect string.
>
> However, this appears not to be the case (see exception below). Is this
> expected?
>
> org.I0Itec.zkclient.exception.ZkException: Unable to connect to <redacted>
> at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:66)
> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:872)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
> at
> kafka.producer.ZKBrokerPartitionInfo.<init>(ZKBrokerPartitionInfo.scala:62)
> at kafka.producer.Producer.<init>(Producer.scala:47)
> at kafka.javaapi.producer.Producer.<init>(Producer.scala:33)
> at kafka.javaapi.producer.Producer.<init>(Producer.scala:40)
> ......
> Caused by: java.net.UnknownHostException: <redacted>
> at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850)
> at
> java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1154)
> at java.net.InetAddress.getAllByName(InetAddress.java:1084)
> at java.net.InetAddress.getAllByName(InetAddress.java:1020)
> at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:382)
> at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:327)
> at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383)
> at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:64)
> ... 19 more
>
> Jason
>