You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 小宇 <mo...@gmail.com> on 2015/04/21 05:22:41 UTC
Kafka server - conflicted ephemeral node
Hi, all
Our kafka server encounter an unexpected shut down last night, I checked
the logs, it says there is a conflicted ephemeral node, but can't find what
issue it is, can anyone help check this, thanks!
[2015-04-21 03:00:00,562] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556241625","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:00:00,564] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:00:08,551] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556241625","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:00:08,553] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:00:22,539] INFO Closing socket connection to /10.144.132.152.
(kafka.network.Processor)
[2015-04-21 03:00:44,228] INFO Closing socket connection to /10.144.38.185.
(kafka.network.Processor)
[2015-04-21 03:00:45,210] INFO Registered broker 0 at path /brokers/ids/0
with address 10.144.38.185:9092. (kafka.utils.ZkUtils$)
[2015-04-21 03:00:45,210] INFO done re-registering broker
(kafka.server.KafkaHealthcheck)
[2015-04-21 03:00:45,210] INFO Subscribing to /brokers/topics path to watch
for new topics (kafka.server.KafkaHealthcheck)
[2015-04-21 03:00:45,212] ERROR Closing socket for /10.144.38.185 because
of error (kafka.network.Processor)
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:524)
at java.nio.channels.SocketChannel.write(SocketChannel.java:493)
at
kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56)
at kafka.network.Processor.write(SocketServer.scala:472)
at kafka.network.Processor.run(SocketServer.scala:342)
at java.lang.Thread.run(Thread.java:745)
[2015-04-21 03:00:45,215] INFO re-registering broker info in ZK for broker
0 (kafka.server.KafkaHealthcheck)
[2015-04-21 03:00:45,223] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:00:45,225] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:00:53,022] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:00:53,888] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:21,366] INFO Registered broker 0 at path /brokers/ids/0
with address 10.144.38.185:9092. (kafka.utils.ZkUtils$)
[2015-04-21 03:01:21,366] INFO done re-registering broker
(kafka.server.KafkaHealthcheck)
[2015-04-21 03:01:21,366] INFO Subscribing to /brokers/topics path to watch
for new topics (kafka.server.KafkaHealthcheck)
[2015-04-21 03:01:21,468] INFO re-registering broker info in ZK for broker
0 (kafka.server.KafkaHealthcheck)
[2015-04-21 03:01:21,471] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:21,472] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:27,476] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:27,477] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:33,481] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:33,483] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:41,589] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:41,590] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:48,398] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:48,405] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:02:19,410] INFO Registered broker 0 at path /brokers/ids/0
with address 10.144.38.185:9092. (kafka.utils.ZkUtils$)
[2015-04-21 03:02:19,410] INFO done re-registering broker
(kafka.server.KafkaHealthcheck)
[2015-04-21 03:02:19,410] INFO Subscribing to /brokers/topics path to watch
for new topics (kafka.server.KafkaHealthcheck)
[2015-04-21 03:02:19,425] INFO re-registering broker info in ZK for broker
0 (kafka.server.KafkaHealthcheck)
[2015-04-21 03:02:19,430] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:02:19,432] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:02:26,128] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:02:26,130] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:02:35,785] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:02:35,786] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
Re: Kafka server - conflicted ephemeral node
Posted by 小宇 <mo...@gmail.com>.
Is there a recommended way to handle this issue?
Thanks!
Mayuresh Gharat <gh...@gmail.com>于2015年4月22日星期三写道：
> This happens due to a bug in zookeeper, sometimes the znode does not get
> deleted automatically.We have seen it many times at Linkedin and are trying
> to investigate further.
>
> Thanks,
>
> Mayuresh
>
> On Mon, Apr 20, 2015 at 8:52 PM, 小宇 <mockingror@gmail.com <javascript:;>>
> wrote:
>
> > Thanks for your response gharatmayuresh1, but I don't know what you mean
> > exactly. I have restart my server and I want to find out the cause in
> case
> > it happen again.
> >
> > 2015-04-21 11:36 GMT+08:00 <gharatmayuresh15@gmail.com <javascript:;>>:
> >
> > > Try bouncing
> > > 10.144.38.185
> > >
> > > This should resolve the issue.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > > Sent from my iPhone
> > >
> > > > On Apr 20, 2015, at 8:22 PM, 小宇 <mockingror@gmail.com <javascript:;>>
> wrote:
> > > >
> > > > 10.144.38.185
> > >
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Kafka server - conflicted ephemeral node
Posted by Mayuresh Gharat <gh...@gmail.com>.
This happens due to a bug in zookeeper, sometimes the znode does not get
deleted automatically.We have seen it many times at Linkedin and are trying
to investigate further.
Thanks,
Mayuresh
On Mon, Apr 20, 2015 at 8:52 PM, 小宇 <mo...@gmail.com> wrote:
> Thanks for your response gharatmayuresh1, but I don't know what you mean
> exactly. I have restart my server and I want to find out the cause in case
> it happen again.
>
> 2015-04-21 11:36 GMT+08:00 <gh...@gmail.com>:
>
> > Try bouncing
> > 10.144.38.185
> >
> > This should resolve the issue.
> >
> > Thanks,
> >
> > Mayuresh
> > Sent from my iPhone
> >
> > > On Apr 20, 2015, at 8:22 PM, 小宇 <mo...@gmail.com> wrote:
> > >
> > > 10.144.38.185
> >
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Kafka server - conflicted ephemeral node
Posted by 小宇 <mo...@gmail.com>.
Thanks for your response gharatmayuresh1, but I don't know what you mean
exactly. I have restart my server and I want to find out the cause in case
it happen again.
2015-04-21 11:36 GMT+08:00 <gh...@gmail.com>:
> Try bouncing
> 10.144.38.185
>
> This should resolve the issue.
>
> Thanks,
>
> Mayuresh
> Sent from my iPhone
>
> > On Apr 20, 2015, at 8:22 PM, 小宇 <mo...@gmail.com> wrote:
> >
> > 10.144.38.185
>
Re: Kafka server - conflicted ephemeral node
Posted by gh...@gmail.com.
Try bouncing
10.144.38.185
This should resolve the issue.
Thanks,
Mayuresh
Sent from my iPhone
> On Apr 20, 2015, at 8:22 PM, 小宇 <mo...@gmail.com> wrote:
>
> 10.144.38.185