You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bongyeon Kim <bo...@gmail.com> on 2014/08/05 04:23:57 UTC
Conflict stored data in Zookeeper
Hi, everyone.
I'm using 0.8.1.1, and I have 8 brokers and 3 topics each have 16
partitions and 3 replicas.
I got unseen logs like below. this is occur every 5 seconds.
[2014-08-05 11:11:32,478] INFO conflict in /brokers/ids/2 data:
{"jmx_port":9992,"timestamp":"1407204339990","host":"172.25.63.9","version":1,"port":9092}
stored data:
{"jmx_port":9992,"timestamp":"1407204133312","host":"172.25.63.9","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2014-08-05 11:11:32,479] INFO I wrote this conflicted ephemeral node
[{"jmx_port":9992,"timestamp":"1407204339990","host":"172.25.63.9","version":1,"port":9092}]
at /brokers/ids/2 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
I hope to know the what makes this messages.
Is it OK that's not ERROR? How can I remove that message?
Thanks in adavnce.
--
*Sincerely*
*,**Bongyeon Kim*
Java Developer & Engineer
Seoul, Korea
Mobile: +82-10-9369-1314
Email: bongyeonkim@gmail.com
Twitter: http://twitter.com/tigerby
Facebook: http://facebook.com/tigerby
Wiki: http://tigerby.com
Re: Conflict stored data in Zookeeper
Posted by Bongyeon Kim <bo...@gmail.com>.
Thanks for the reply.
Before I seen that log, I produced a lot of events for performance test.
(approximately 3G/min), and I have seen that log in an hour or two. and
I've got ERROR meesage frequently like below.
[2014-08-04 11:27:54,547] ERROR [ReplicaFetcherThread-0-6], Error in fetch
Name: FetchRequest; Version: 0; CorrelationId: 6263874; ClientId:
ReplicaFetcherThread-0-6; ReplicaId: 2; MaxWait: 500 ms; MinBytes: 1 bytes;
RequestInfo: [topicTRACE,13] ->
PartitionFetchInfo(10091892,1073741824),[topicTRACE,1] ->
PartitionFetchInfo(10174056,1073741824),[topicTRACE,9] ->
PartitionFetchInfo(10087558,1073741824),[topicTRACE,5] ->
PartitionFetchInfo(10148805,1073741824) (kafka.server.ReplicaFetcherThread)
java.io.EOFException: Received -1 when reading from channel, socket has
likely been closed.
While I was doing performance test, replications was getting shrinked, and
when it's done, I got my replications back. then I did preferred replica
election.
That's all I've done. I never start/stop brokers since I started it first
time.
Anyway, I'll try to restart my brokers like you said.
Thanks.
On Tue, Aug 5, 2014 at 1:26 PM, Joe Stein <jo...@stealth.ly> wrote:
> I have seen an issue similar to this but with the /controller node.
>
> I am going to update https://issues.apache.org/jira/browse/KAFKA-1387 with
> the steps to reproduce the issue I ran into right now.
>
> I don't know what steps caused what you ran into it is very odd that
> shouldn't happen.
>
> Were you doing anything with the cluster before this happened?
> starting/stopping nodes? any steps that might help reproduce it? any more
> info in any logs?
>
> I would recommend shutting down that node, make sure the znode is gone and
> starting it back up again.
>
> I also agree INFO is not a good status in that function, it is in a
> while(true) loop that may never end should at least be a WARN.
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
>
> On Mon, Aug 4, 2014 at 10:23 PM, Bongyeon Kim <bo...@gmail.com>
> wrote:
>
> > Hi, everyone.
> >
> > I'm using 0.8.1.1, and I have 8 brokers and 3 topics each have 16
> > partitions and 3 replicas.
> >
> > I got unseen logs like below. this is occur every 5 seconds.
> >
> >
> > [2014-08-05 11:11:32,478] INFO conflict in /brokers/ids/2 data:
> >
> >
> {"jmx_port":9992,"timestamp":"1407204339990","host":"172.25.63.9","version":1,"port":9092}
> > stored data:
> >
> >
> {"jmx_port":9992,"timestamp":"1407204133312","host":"172.25.63.9","version":1,"port":9092}
> > (kafka.utils.ZkUtils$)
> > [2014-08-05 11:11:32,479] INFO I wrote this conflicted ephemeral node
> >
> >
> [{"jmx_port":9992,"timestamp":"1407204339990","host":"172.25.63.9","version":1,"port":9092}]
> > at /brokers/ids/2 a while back in a different session, hence I will
> backoff
> > for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
> >
> >
> > I hope to know the what makes this messages.
> > Is it OK that's not ERROR? How can I remove that message?
> >
> >
> > Thanks in adavnce.
> >
> > --
> > *Sincerely*
> > *,**Bongyeon Kim*
> >
> > Java Developer & Engineer
> > Seoul, Korea
> > Mobile: +82-10-9369-1314
> > Email: bongyeonkim@gmail.com
> > Twitter: http://twitter.com/tigerby
> > Facebook: http://facebook.com/tigerby
> > Wiki: http://tigerby.com
> >
>
--
*Sincerely*
*,**Bongyeon Kim*
Java Developer & Engineer
Seoul, Korea
Mobile: +82-10-9369-1314
Email: bongyeonkim@gmail.com
Twitter: http://twitter.com/tigerby
Facebook: http://facebook.com/tigerby
Wiki: http://tigerby.com
Re: Conflict stored data in Zookeeper
Posted by Joe Stein <jo...@stealth.ly>.
I have seen an issue similar to this but with the /controller node.
I am going to update https://issues.apache.org/jira/browse/KAFKA-1387 with
the steps to reproduce the issue I ran into right now.
I don't know what steps caused what you ran into it is very odd that
shouldn't happen.
Were you doing anything with the cluster before this happened?
starting/stopping nodes? any steps that might help reproduce it? any more
info in any logs?
I would recommend shutting down that node, make sure the znode is gone and
starting it back up again.
I also agree INFO is not a good status in that function, it is in a
while(true) loop that may never end should at least be a WARN.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Mon, Aug 4, 2014 at 10:23 PM, Bongyeon Kim <bo...@gmail.com>
wrote:
> Hi, everyone.
>
> I'm using 0.8.1.1, and I have 8 brokers and 3 topics each have 16
> partitions and 3 replicas.
>
> I got unseen logs like below. this is occur every 5 seconds.
>
>
> [2014-08-05 11:11:32,478] INFO conflict in /brokers/ids/2 data:
>
> {"jmx_port":9992,"timestamp":"1407204339990","host":"172.25.63.9","version":1,"port":9092}
> stored data:
>
> {"jmx_port":9992,"timestamp":"1407204133312","host":"172.25.63.9","version":1,"port":9092}
> (kafka.utils.ZkUtils$)
> [2014-08-05 11:11:32,479] INFO I wrote this conflicted ephemeral node
>
> [{"jmx_port":9992,"timestamp":"1407204339990","host":"172.25.63.9","version":1,"port":9092}]
> at /brokers/ids/2 a while back in a different session, hence I will backoff
> for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
>
>
> I hope to know the what makes this messages.
> Is it OK that's not ERROR? How can I remove that message?
>
>
> Thanks in adavnce.
>
> --
> *Sincerely*
> *,**Bongyeon Kim*
>
> Java Developer & Engineer
> Seoul, Korea
> Mobile: +82-10-9369-1314
> Email: bongyeonkim@gmail.com
> Twitter: http://twitter.com/tigerby
> Facebook: http://facebook.com/tigerby
> Wiki: http://tigerby.com
>