You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Joe San <co...@gmail.com> on 2016/02/15 21:22:45 UTC
Enable Kafka Consumer 0.8.2.1 Reconnect to Zookeeper
Any ideas as to which property should I set to enable Zookeeper
re-connection? I have the following properties defined for my consumer
(High Level Consumer API). Is this enough for a automatic Zookeeper
re-connect?
val props = new Properties()
props.put("zookeeper.connect", zookeeper)
props.put("group.id", groupId)
props.put("auto.commit.enabled", "false")
// this timeout is needed so that we do not block on the stream!
props.put("consumer.timeout.ms", "1")
props.put("zookeeper.sync.time.ms", "200")
Re: Enable Kafka Consumer 0.8.2.1 Reconnect to Zookeeper
Posted by Alexis Midon <al...@airbnb.com.INVALID>.
A Connection and a ZK session are two different things.
The ZK master keeps track of a client session validity. When a client
connection gets interrupted, its associated session goes into Disconnected
state, after a while it will be Expired, but if a new connection is
established before the timeout then the session will be Connected again.
see
http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkSessions
Example state transitions for an expired session as seen by the expired
session's watcher:
1.
'connected' : session is established and client is communicating with
cluster (client/server communication is operating properly)
2.
.... client is partitioned from the cluster
3.
'disconnected' : client has lost connectivity with the cluster
4.
.... time elapses, after 'timeout' period the cluster expires the
session, nothing is seen by client as it is disconnected from cluster
5.
.... time elapses, the client regains network level connectivity with
the cluster
6.
'expired' : eventually the client reconnects to the cluster, it is then
notified of the expiration
On Wed, Feb 17, 2016 at 11:38 AM Christian Posta <ch...@gmail.com>
wrote:
> I believe so. Happy to be corrected.
>
> On Wed, Feb 17, 2016 at 12:31 PM, Joe San <co...@gmail.com> wrote:
>
> > So if I use the High Level Consumer API, using the ConsumerConnector, I
> get
> > this automatic zookeeper connection for free?
> >
> > On Wed, Feb 17, 2016 at 8:25 PM, Christian Posta <
> > christian.posta@gmail.com>
> > wrote:
> >
> > > i believe reconnect is handled automatically by the client... is that
> > what
> > > you're asking
> > > peek here to see how it does that and when:
> > >
> > >
> > >
> >
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/ClientCnxn.java#L1153
> > >
> > > On Wed, Feb 17, 2016 at 12:14 PM, Joe San <co...@gmail.com>
> > wrote:
> > >
> > > > It is all pretty strange. Here is what I see in my logs as soon as I
> > > > voluntarily shutdown Zookeeper!
> > > >
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > ~[na:1.8.0_60]
> > > > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> > > > ~[na:1.8.0_60]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> > > > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > > > at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> > > > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > > > 20160217-20:12:44.960+0100
> > > > [sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
> > > > 127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening
> > socket
> > > > connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to
> > > > authenticate using SASL (unknown error)
> > > > 20160217-20:12:44.960+0100
> > > > [sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
> > > > 127.0.0.1:2181)] WARN org.apache.zookeeper.ClientCnxn - Session
> > > > 0x152ea19656b005c for server null, unexpected error, closing socket
> > > > connection and attempting reconnect
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > ~[na:1.8.0_60]
> > > > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> > > > ~[na:1.8.0_60]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> > > > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > > > at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> > > > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > > >
> > > > It just keep repeating trying to reconnect for ever! So I just wanted
> > to
> > > > know which property from my setting in my email above is responsible
> > for
> > > > this auto reconnect mechanism?
> > > >
> > > > On Wed, Feb 17, 2016 at 8:04 PM, Christian Posta <
> > > > christian.posta@gmail.com>
> > > > wrote:
> > > >
> > > > > Yep, assuming you haven't completely partitioned that client from
> the
> > > > > cluster, ZK should automatically try to connect/reconnect to other
> > > peers
> > > > in
> > > > > the server list. Otherwise, it's as Alexis said -- your session
> would
> > > > > expire; you'd have to recreate the session once you have
> connectivity
> > > > >
> > > > > On Wed, Feb 17, 2016 at 2:30 AM, Alexis Midon <
> > > > > alexis.midon@airbnb.com.invalid> wrote:
> > > > >
> > > > > > By "re-connect", I'm assuming that the ZK session is expired, not
> > > > > > disconnected.
> > > > > > For details see
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkSessions
> > > > > >
> > > > > > In that case, the high level consumer is basically dead, and the
> > > > > > application should create a new instance of it.
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 15, 2016 at 12:22 PM Joe San <
> codeintheopen@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Any ideas as to which property should I set to enable Zookeeper
> > > > > > > re-connection? I have the following properties defined for my
> > > > consumer
> > > > > > > (High Level Consumer API). Is this enough for a automatic
> > Zookeeper
> > > > > > > re-connect?
> > > > > > >
> > > > > > > val props = new Properties()
> > > > > > > props.put("zookeeper.connect", zookeeper)
> > > > > > > props.put("group.id", groupId)
> > > > > > > props.put("auto.commit.enabled", "false")
> > > > > > > // this timeout is needed so that we do not block on the
> stream!
> > > > > > > props.put("consumer.timeout.ms", "1")
> > > > > > > props.put("zookeeper.sync.time.ms", "200")
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Christian Posta*
> > > > > twitter: @christianposta
> > > > > http://www.christianposta.com/blog
> > > > > http://fabric8.io
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Christian Posta*
> > > twitter: @christianposta
> > > http://www.christianposta.com/blog
> > > http://fabric8.io
> > >
> >
>
>
>
> --
> *Christian Posta*
> twitter: @christianposta
> http://www.christianposta.com/blog
> http://fabric8.io
>
Re: Enable Kafka Consumer 0.8.2.1 Reconnect to Zookeeper
Posted by Christian Posta <ch...@gmail.com>.
I believe so. Happy to be corrected.
On Wed, Feb 17, 2016 at 12:31 PM, Joe San <co...@gmail.com> wrote:
> So if I use the High Level Consumer API, using the ConsumerConnector, I get
> this automatic zookeeper connection for free?
>
> On Wed, Feb 17, 2016 at 8:25 PM, Christian Posta <
> christian.posta@gmail.com>
> wrote:
>
> > i believe reconnect is handled automatically by the client... is that
> what
> > you're asking
> > peek here to see how it does that and when:
> >
> >
> >
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/ClientCnxn.java#L1153
> >
> > On Wed, Feb 17, 2016 at 12:14 PM, Joe San <co...@gmail.com>
> wrote:
> >
> > > It is all pretty strange. Here is what I see in my logs as soon as I
> > > voluntarily shutdown Zookeeper!
> > >
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > ~[na:1.8.0_60]
> > > at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> > > ~[na:1.8.0_60]
> > > at
> > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> > > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> > > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > > 20160217-20:12:44.960+0100
> > > [sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
> > > 127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening
> socket
> > > connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to
> > > authenticate using SASL (unknown error)
> > > 20160217-20:12:44.960+0100
> > > [sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
> > > 127.0.0.1:2181)] WARN org.apache.zookeeper.ClientCnxn - Session
> > > 0x152ea19656b005c for server null, unexpected error, closing socket
> > > connection and attempting reconnect
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > ~[na:1.8.0_60]
> > > at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> > > ~[na:1.8.0_60]
> > > at
> > >
> > >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> > > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> > > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > >
> > > It just keep repeating trying to reconnect for ever! So I just wanted
> to
> > > know which property from my setting in my email above is responsible
> for
> > > this auto reconnect mechanism?
> > >
> > > On Wed, Feb 17, 2016 at 8:04 PM, Christian Posta <
> > > christian.posta@gmail.com>
> > > wrote:
> > >
> > > > Yep, assuming you haven't completely partitioned that client from the
> > > > cluster, ZK should automatically try to connect/reconnect to other
> > peers
> > > in
> > > > the server list. Otherwise, it's as Alexis said -- your session would
> > > > expire; you'd have to recreate the session once you have connectivity
> > > >
> > > > On Wed, Feb 17, 2016 at 2:30 AM, Alexis Midon <
> > > > alexis.midon@airbnb.com.invalid> wrote:
> > > >
> > > > > By "re-connect", I'm assuming that the ZK session is expired, not
> > > > > disconnected.
> > > > > For details see
> > > > >
> > > > >
> > > >
> > >
> >
> http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkSessions
> > > > >
> > > > > In that case, the high level consumer is basically dead, and the
> > > > > application should create a new instance of it.
> > > > >
> > > > >
> > > > > On Mon, Feb 15, 2016 at 12:22 PM Joe San <co...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Any ideas as to which property should I set to enable Zookeeper
> > > > > > re-connection? I have the following properties defined for my
> > > consumer
> > > > > > (High Level Consumer API). Is this enough for a automatic
> Zookeeper
> > > > > > re-connect?
> > > > > >
> > > > > > val props = new Properties()
> > > > > > props.put("zookeeper.connect", zookeeper)
> > > > > > props.put("group.id", groupId)
> > > > > > props.put("auto.commit.enabled", "false")
> > > > > > // this timeout is needed so that we do not block on the stream!
> > > > > > props.put("consumer.timeout.ms", "1")
> > > > > > props.put("zookeeper.sync.time.ms", "200")
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Christian Posta*
> > > > twitter: @christianposta
> > > > http://www.christianposta.com/blog
> > > > http://fabric8.io
> > > >
> > >
> >
> >
> >
> > --
> > *Christian Posta*
> > twitter: @christianposta
> > http://www.christianposta.com/blog
> > http://fabric8.io
> >
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
Re: Enable Kafka Consumer 0.8.2.1 Reconnect to Zookeeper
Posted by Joe San <co...@gmail.com>.
So if I use the High Level Consumer API, using the ConsumerConnector, I get
this automatic zookeeper connection for free?
On Wed, Feb 17, 2016 at 8:25 PM, Christian Posta <ch...@gmail.com>
wrote:
> i believe reconnect is handled automatically by the client... is that what
> you're asking
> peek here to see how it does that and when:
>
>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/ClientCnxn.java#L1153
>
> On Wed, Feb 17, 2016 at 12:14 PM, Joe San <co...@gmail.com> wrote:
>
> > It is all pretty strange. Here is what I see in my logs as soon as I
> > voluntarily shutdown Zookeeper!
> >
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> ~[na:1.8.0_60]
> > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> > ~[na:1.8.0_60]
> > at
> >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > 20160217-20:12:44.960+0100
> > [sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
> > 127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket
> > connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 20160217-20:12:44.960+0100
> > [sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
> > 127.0.0.1:2181)] WARN org.apache.zookeeper.ClientCnxn - Session
> > 0x152ea19656b005c for server null, unexpected error, closing socket
> > connection and attempting reconnect
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> ~[na:1.8.0_60]
> > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> > ~[na:1.8.0_60]
> > at
> >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> > ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> >
> > It just keep repeating trying to reconnect for ever! So I just wanted to
> > know which property from my setting in my email above is responsible for
> > this auto reconnect mechanism?
> >
> > On Wed, Feb 17, 2016 at 8:04 PM, Christian Posta <
> > christian.posta@gmail.com>
> > wrote:
> >
> > > Yep, assuming you haven't completely partitioned that client from the
> > > cluster, ZK should automatically try to connect/reconnect to other
> peers
> > in
> > > the server list. Otherwise, it's as Alexis said -- your session would
> > > expire; you'd have to recreate the session once you have connectivity
> > >
> > > On Wed, Feb 17, 2016 at 2:30 AM, Alexis Midon <
> > > alexis.midon@airbnb.com.invalid> wrote:
> > >
> > > > By "re-connect", I'm assuming that the ZK session is expired, not
> > > > disconnected.
> > > > For details see
> > > >
> > > >
> > >
> >
> http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkSessions
> > > >
> > > > In that case, the high level consumer is basically dead, and the
> > > > application should create a new instance of it.
> > > >
> > > >
> > > > On Mon, Feb 15, 2016 at 12:22 PM Joe San <co...@gmail.com>
> > > wrote:
> > > >
> > > > > Any ideas as to which property should I set to enable Zookeeper
> > > > > re-connection? I have the following properties defined for my
> > consumer
> > > > > (High Level Consumer API). Is this enough for a automatic Zookeeper
> > > > > re-connect?
> > > > >
> > > > > val props = new Properties()
> > > > > props.put("zookeeper.connect", zookeeper)
> > > > > props.put("group.id", groupId)
> > > > > props.put("auto.commit.enabled", "false")
> > > > > // this timeout is needed so that we do not block on the stream!
> > > > > props.put("consumer.timeout.ms", "1")
> > > > > props.put("zookeeper.sync.time.ms", "200")
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Christian Posta*
> > > twitter: @christianposta
> > > http://www.christianposta.com/blog
> > > http://fabric8.io
> > >
> >
>
>
>
> --
> *Christian Posta*
> twitter: @christianposta
> http://www.christianposta.com/blog
> http://fabric8.io
>
Re: Enable Kafka Consumer 0.8.2.1 Reconnect to Zookeeper
Posted by Christian Posta <ch...@gmail.com>.
i believe reconnect is handled automatically by the client... is that what
you're asking
peek here to see how it does that and when:
https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/ClientCnxn.java#L1153
On Wed, Feb 17, 2016 at 12:14 PM, Joe San <co...@gmail.com> wrote:
> It is all pretty strange. Here is what I see in my logs as soon as I
> voluntarily shutdown Zookeeper!
>
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_60]
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ~[na:1.8.0_60]
> at
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> 20160217-20:12:44.960+0100
> [sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
> 127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket
> connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 20160217-20:12:44.960+0100
> [sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
> 127.0.0.1:2181)] WARN org.apache.zookeeper.ClientCnxn - Session
> 0x152ea19656b005c for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_60]
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> ~[na:1.8.0_60]
> at
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
>
> It just keep repeating trying to reconnect for ever! So I just wanted to
> know which property from my setting in my email above is responsible for
> this auto reconnect mechanism?
>
> On Wed, Feb 17, 2016 at 8:04 PM, Christian Posta <
> christian.posta@gmail.com>
> wrote:
>
> > Yep, assuming you haven't completely partitioned that client from the
> > cluster, ZK should automatically try to connect/reconnect to other peers
> in
> > the server list. Otherwise, it's as Alexis said -- your session would
> > expire; you'd have to recreate the session once you have connectivity
> >
> > On Wed, Feb 17, 2016 at 2:30 AM, Alexis Midon <
> > alexis.midon@airbnb.com.invalid> wrote:
> >
> > > By "re-connect", I'm assuming that the ZK session is expired, not
> > > disconnected.
> > > For details see
> > >
> > >
> >
> http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkSessions
> > >
> > > In that case, the high level consumer is basically dead, and the
> > > application should create a new instance of it.
> > >
> > >
> > > On Mon, Feb 15, 2016 at 12:22 PM Joe San <co...@gmail.com>
> > wrote:
> > >
> > > > Any ideas as to which property should I set to enable Zookeeper
> > > > re-connection? I have the following properties defined for my
> consumer
> > > > (High Level Consumer API). Is this enough for a automatic Zookeeper
> > > > re-connect?
> > > >
> > > > val props = new Properties()
> > > > props.put("zookeeper.connect", zookeeper)
> > > > props.put("group.id", groupId)
> > > > props.put("auto.commit.enabled", "false")
> > > > // this timeout is needed so that we do not block on the stream!
> > > > props.put("consumer.timeout.ms", "1")
> > > > props.put("zookeeper.sync.time.ms", "200")
> > > >
> > >
> >
> >
> >
> > --
> > *Christian Posta*
> > twitter: @christianposta
> > http://www.christianposta.com/blog
> > http://fabric8.io
> >
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
Re: Enable Kafka Consumer 0.8.2.1 Reconnect to Zookeeper
Posted by Joe San <co...@gmail.com>.
It is all pretty strange. Here is what I see in my logs as soon as I
voluntarily shutdown Zookeeper!
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_60]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[na:1.8.0_60]
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
20160217-20:12:44.960+0100
[sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket
connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to
authenticate using SASL (unknown error)
20160217-20:12:44.960+0100
[sally-kafka-consumer-akka.actor.default-dispatcher-4-SendThread(
127.0.0.1:2181)] WARN org.apache.zookeeper.ClientCnxn - Session
0x152ea19656b005c for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_60]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[na:1.8.0_60]
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
It just keep repeating trying to reconnect for ever! So I just wanted to
know which property from my setting in my email above is responsible for
this auto reconnect mechanism?
On Wed, Feb 17, 2016 at 8:04 PM, Christian Posta <ch...@gmail.com>
wrote:
> Yep, assuming you haven't completely partitioned that client from the
> cluster, ZK should automatically try to connect/reconnect to other peers in
> the server list. Otherwise, it's as Alexis said -- your session would
> expire; you'd have to recreate the session once you have connectivity
>
> On Wed, Feb 17, 2016 at 2:30 AM, Alexis Midon <
> alexis.midon@airbnb.com.invalid> wrote:
>
> > By "re-connect", I'm assuming that the ZK session is expired, not
> > disconnected.
> > For details see
> >
> >
> http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkSessions
> >
> > In that case, the high level consumer is basically dead, and the
> > application should create a new instance of it.
> >
> >
> > On Mon, Feb 15, 2016 at 12:22 PM Joe San <co...@gmail.com>
> wrote:
> >
> > > Any ideas as to which property should I set to enable Zookeeper
> > > re-connection? I have the following properties defined for my consumer
> > > (High Level Consumer API). Is this enough for a automatic Zookeeper
> > > re-connect?
> > >
> > > val props = new Properties()
> > > props.put("zookeeper.connect", zookeeper)
> > > props.put("group.id", groupId)
> > > props.put("auto.commit.enabled", "false")
> > > // this timeout is needed so that we do not block on the stream!
> > > props.put("consumer.timeout.ms", "1")
> > > props.put("zookeeper.sync.time.ms", "200")
> > >
> >
>
>
>
> --
> *Christian Posta*
> twitter: @christianposta
> http://www.christianposta.com/blog
> http://fabric8.io
>
Re: Enable Kafka Consumer 0.8.2.1 Reconnect to Zookeeper
Posted by Christian Posta <ch...@gmail.com>.
Yep, assuming you haven't completely partitioned that client from the
cluster, ZK should automatically try to connect/reconnect to other peers in
the server list. Otherwise, it's as Alexis said -- your session would
expire; you'd have to recreate the session once you have connectivity
On Wed, Feb 17, 2016 at 2:30 AM, Alexis Midon <
alexis.midon@airbnb.com.invalid> wrote:
> By "re-connect", I'm assuming that the ZK session is expired, not
> disconnected.
> For details see
>
> http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkSessions
>
> In that case, the high level consumer is basically dead, and the
> application should create a new instance of it.
>
>
> On Mon, Feb 15, 2016 at 12:22 PM Joe San <co...@gmail.com> wrote:
>
> > Any ideas as to which property should I set to enable Zookeeper
> > re-connection? I have the following properties defined for my consumer
> > (High Level Consumer API). Is this enough for a automatic Zookeeper
> > re-connect?
> >
> > val props = new Properties()
> > props.put("zookeeper.connect", zookeeper)
> > props.put("group.id", groupId)
> > props.put("auto.commit.enabled", "false")
> > // this timeout is needed so that we do not block on the stream!
> > props.put("consumer.timeout.ms", "1")
> > props.put("zookeeper.sync.time.ms", "200")
> >
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
Re: Enable Kafka Consumer 0.8.2.1 Reconnect to Zookeeper
Posted by Alexis Midon <al...@airbnb.com.INVALID>.
By "re-connect", I'm assuming that the ZK session is expired, not
disconnected.
For details see
http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkSessions
In that case, the high level consumer is basically dead, and the
application should create a new instance of it.
On Mon, Feb 15, 2016 at 12:22 PM Joe San <co...@gmail.com> wrote:
> Any ideas as to which property should I set to enable Zookeeper
> re-connection? I have the following properties defined for my consumer
> (High Level Consumer API). Is this enough for a automatic Zookeeper
> re-connect?
>
> val props = new Properties()
> props.put("zookeeper.connect", zookeeper)
> props.put("group.id", groupId)
> props.put("auto.commit.enabled", "false")
> // this timeout is needed so that we do not block on the stream!
> props.put("consumer.timeout.ms", "1")
> props.put("zookeeper.sync.time.ms", "200")
>