You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Srividhya Anantharamakrishnan <sr...@hedviginc.com> on 2015/06/23 00:50:46 UTC
kafka.common.NotLeaderForPartitionException in Consumer
Hi,
I have the following Kafka Set up - Two 3-node clusters A, B, where each
node is acting as a broker and is connected by on ZK running in one node in
cluster A.
I was able to publish messages from cluster A and could subscribe to
consuming from both A and B clusters.
However, I am suddenly running into the following issue:
[ConsumerFetcherThread-group1_<NodeName>-1435011736956-12cac90d-0-2] ERROR
[kafka.consumer.ConsumerFetcherThread] (Logging.scala:97) -
[ConsumerFetcherThread-group1_<NodeName>-1435011736956-12cac90d-0-2], Error
for partition [TestTop,0] to broker 2:class
kafka.common.NotLeaderForPartitionException
I did ensure that all the 3 brokers are running. I have the following
consumer properties:
Properties props = new Properties();
props.put("zookeeper.connect", "192.168.111.203:2181");
props.put("group.id", "group1");
props.put("zookeeper.session.timeout.ms", "6000");
props.put("zookeeper.sync.time.ms", "2000");
props.put("auto.commit.enable", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("auto.offset.reset", "smallest");
props.put("consumer.timeout.ms", "120000");
props.put("backoff.increment.ms", "1500");
Any pointers would really help!
Thanks!
Re: kafka.common.NotLeaderForPartitionException in Consumer
Posted by Jason Gustafson <ja...@confluent.io>.
I think this exception is logged by the consumer when there is a leadership
change for a partition (which might be caused by one of the brokers going
down or losing their session with zookeeper). Typically the consumer will
recover after the cluster finds a new leader and the consumer updates its
metadata. Are you not seeing that?
On Mon, Jun 22, 2015 at 5:05 PM, Srividhya Anantharamakrishnan <
srividhya@hedviginc.com> wrote:
> ok, let me make it a little more clear.
>
> Datacenter A has 3 nodes, each acting as a broker, publishing messages to
> one of the nodes that has zookeeper running.
>
> Datacenter B has the same set up.
>
> Now, I am trying to publish message from one of the nodes in A to the ZK in
> A and make one of the nodes in B consume the message by connecting to A's
> ZK.
>
>
>
> On Mon, Jun 22, 2015 at 4:25 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hi Srividhya,
> >
> > I'm a little confused about your setup. You have both clusters pointed to
> > the same zookeeper, right? You don't appear to be using the zookeeper
> > chroot option, so I think they would just form a single cluster.
> >
> > -Jason
> >
> > On Mon, Jun 22, 2015 at 3:50 PM, Srividhya Anantharamakrishnan <
> > srividhya@hedviginc.com> wrote:
> >
> > > Hi,
> > >
> > > I have the following Kafka Set up - Two 3-node clusters A, B, where
> each
> > > node is acting as a broker and is connected by on ZK running in one
> node
> > in
> > > cluster A.
> > >
> > > I was able to publish messages from cluster A and could subscribe to
> > > consuming from both A and B clusters.
> > >
> > > However, I am suddenly running into the following issue:
> > >
> > > [ConsumerFetcherThread-group1_<NodeName>-1435011736956-12cac90d-0-2]
> > ERROR
> > > [kafka.consumer.ConsumerFetcherThread] (Logging.scala:97) -
> > > [ConsumerFetcherThread-group1_<NodeName>-1435011736956-12cac90d-0-2],
> > Error
> > > for partition [TestTop,0] to broker 2:class
> > > kafka.common.NotLeaderForPartitionException
> > >
> > > I did ensure that all the 3 brokers are running. I have the following
> > > consumer properties:
> > >
> > > Properties props = new Properties();
> > >
> > > props.put("zookeeper.connect", "192.168.111.203:2181");
> > >
> > > props.put("group.id", "group1");
> > >
> > > props.put("zookeeper.session.timeout.ms", "6000");
> > >
> > > props.put("zookeeper.sync.time.ms", "2000");
> > >
> > > props.put("auto.commit.enable", "true");
> > >
> > > props.put("auto.commit.interval.ms", "1000");
> > >
> > > props.put("auto.offset.reset", "smallest");
> > >
> > > props.put("consumer.timeout.ms", "120000");
> > >
> > > props.put("backoff.increment.ms", "1500");
> > > Any pointers would really help!
> > >
> > > Thanks!
> > >
> >
>
Re: kafka.common.NotLeaderForPartitionException in Consumer
Posted by Srividhya Anantharamakrishnan <sr...@hedviginc.com>.
ok, let me make it a little more clear.
Datacenter A has 3 nodes, each acting as a broker, publishing messages to
one of the nodes that has zookeeper running.
Datacenter B has the same set up.
Now, I am trying to publish message from one of the nodes in A to the ZK in
A and make one of the nodes in B consume the message by connecting to A's
ZK.
On Mon, Jun 22, 2015 at 4:25 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hi Srividhya,
>
> I'm a little confused about your setup. You have both clusters pointed to
> the same zookeeper, right? You don't appear to be using the zookeeper
> chroot option, so I think they would just form a single cluster.
>
> -Jason
>
> On Mon, Jun 22, 2015 at 3:50 PM, Srividhya Anantharamakrishnan <
> srividhya@hedviginc.com> wrote:
>
> > Hi,
> >
> > I have the following Kafka Set up - Two 3-node clusters A, B, where each
> > node is acting as a broker and is connected by on ZK running in one node
> in
> > cluster A.
> >
> > I was able to publish messages from cluster A and could subscribe to
> > consuming from both A and B clusters.
> >
> > However, I am suddenly running into the following issue:
> >
> > [ConsumerFetcherThread-group1_<NodeName>-1435011736956-12cac90d-0-2]
> ERROR
> > [kafka.consumer.ConsumerFetcherThread] (Logging.scala:97) -
> > [ConsumerFetcherThread-group1_<NodeName>-1435011736956-12cac90d-0-2],
> Error
> > for partition [TestTop,0] to broker 2:class
> > kafka.common.NotLeaderForPartitionException
> >
> > I did ensure that all the 3 brokers are running. I have the following
> > consumer properties:
> >
> > Properties props = new Properties();
> >
> > props.put("zookeeper.connect", "192.168.111.203:2181");
> >
> > props.put("group.id", "group1");
> >
> > props.put("zookeeper.session.timeout.ms", "6000");
> >
> > props.put("zookeeper.sync.time.ms", "2000");
> >
> > props.put("auto.commit.enable", "true");
> >
> > props.put("auto.commit.interval.ms", "1000");
> >
> > props.put("auto.offset.reset", "smallest");
> >
> > props.put("consumer.timeout.ms", "120000");
> >
> > props.put("backoff.increment.ms", "1500");
> > Any pointers would really help!
> >
> > Thanks!
> >
>
Re: kafka.common.NotLeaderForPartitionException in Consumer
Posted by Jason Gustafson <ja...@confluent.io>.
Hi Srividhya,
I'm a little confused about your setup. You have both clusters pointed to
the same zookeeper, right? You don't appear to be using the zookeeper
chroot option, so I think they would just form a single cluster.
-Jason
On Mon, Jun 22, 2015 at 3:50 PM, Srividhya Anantharamakrishnan <
srividhya@hedviginc.com> wrote:
> Hi,
>
> I have the following Kafka Set up - Two 3-node clusters A, B, where each
> node is acting as a broker and is connected by on ZK running in one node in
> cluster A.
>
> I was able to publish messages from cluster A and could subscribe to
> consuming from both A and B clusters.
>
> However, I am suddenly running into the following issue:
>
> [ConsumerFetcherThread-group1_<NodeName>-1435011736956-12cac90d-0-2] ERROR
> [kafka.consumer.ConsumerFetcherThread] (Logging.scala:97) -
> [ConsumerFetcherThread-group1_<NodeName>-1435011736956-12cac90d-0-2], Error
> for partition [TestTop,0] to broker 2:class
> kafka.common.NotLeaderForPartitionException
>
> I did ensure that all the 3 brokers are running. I have the following
> consumer properties:
>
> Properties props = new Properties();
>
> props.put("zookeeper.connect", "192.168.111.203:2181");
>
> props.put("group.id", "group1");
>
> props.put("zookeeper.session.timeout.ms", "6000");
>
> props.put("zookeeper.sync.time.ms", "2000");
>
> props.put("auto.commit.enable", "true");
>
> props.put("auto.commit.interval.ms", "1000");
>
> props.put("auto.offset.reset", "smallest");
>
> props.put("consumer.timeout.ms", "120000");
>
> props.put("backoff.increment.ms", "1500");
> Any pointers would really help!
>
> Thanks!
>