You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nomar Morado <no...@gmail.com> on 2016/08/29 12:12:26 UTC
a broker is already registered on path /brokers/ids/1
we would get this occasionally after a weekend reboot/restart.
we tried restarting a couple of times all to naught.
we had to delete dk's directory to get his going again.
any ideas what might cause this issue and suggestions on how to resolve
this?
thanks.
Re: a broker is already registered on path /brokers/ids/1
Posted by Nomar Morado <j3...@icloud.com>.
Kafka broker in my case
Printing e-mails wastes valuable natural resources. Please don't print this message unless it is absolutely necessary. Thank you for thinking green!
Sent from my iPhone
> On Aug 30, 2016, at 1:19 PM, Flavio Junqueira <fp...@apache.org> wrote:
>
> Is the zookeeper process you're referring to the server or the client (broker in the case of Kafka)?
>
> If you're referring to the server, then the ensemble recovers from the disk state of the servers. If it is a follower, then there is nothing to recover, the leader already has all the necessary data. If it is the leader, then it will get the ephemeral/session data from disk.
>
> If you're referring to the Kafka broker, then you should be seeing a message like this in your log in the case the broker finds a znode upon registration:
>
> case e: ZkNodeExistsException =>
> throw new RuntimeException("A broker is already registered on the path " + brokerIdPath
> + ". This probably " + "indicates that you either have configured a brokerid that is already in use, or "
> + "else you have shutdown this broker and restarted it faster than the zookeeper "
> + "timeout so it appears to be re-registering.")
>
>
> Does it help?
>
> -Flavio
>
>
>> On 30 Aug 2016, at 14:35, J316 Services <j3...@icloud.com> wrote:
>>
>> Setup is 3 participants + 1 observer.
>>
>> We used 3.5.0 for dynamic configs.
>>
>> What happens when a system forced killed the zookeeper process and it did not have time to clean up ephemeral? Does it recover in that scenario?
>>
>>
>> Thanks.
>>
>> Sent from my iPad
>>
>>> On Aug 30, 2016, at 8:53 AM, Flavio Junqueira <fp...@apache.org> wrote:
>>>
>>> I think you're saying that the session isn't expiring that and the ephemeral isn't getting deleted. Or maybe the session is expiring but the ephemeral isn't being deleted? It'd be great if you could check that the broker session is expiring eventually.
>>>
>>> Since you're on the 3.5 branch, you may want to use 3.5.2-alpha or simply try 3.4.8 as Harsha suggested.
>>>
>>> -Flavio
>>>
>>>> On 29 Aug 2016, at 18:44, Harsha Chintalapani <ka...@harsha.io> wrote:
>>>>
>>>> how many brokers you've in this cluster. Do you try using a stable
>>>> zookeeper release like 3.4.8?
>>>> -Harhsa
>>>>
>>>>> On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
>>>>>
>>>>> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>>>>>
>>>>> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> we would get this occasionally after a weekend reboot/restart.
>>>>>>
>>>>>> we tried restarting a couple of times all to naught.
>>>>>>
>>>>>> we had to delete dk's directory to get his going again.
>>>>>>
>>>>>> any ideas what might cause this issue and suggestions on how to resolve
>>>>>> this?
>>>>>>
>>>>>>
>>>>>> thanks.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Nomar Morado
>>>
>
Re: a broker is already registered on path /brokers/ids/1
Posted by Nomar Morado <j3...@icloud.com>.
Kafka broker in my case
Printing e-mails wastes valuable natural resources. Please don't print this message unless it is absolutely necessary. Thank you for thinking green!
Sent from my iPhone
> On Aug 30, 2016, at 1:19 PM, Flavio Junqueira <fp...@apache.org> wrote:
>
> Is the zookeeper process you're referring to the server or the client (broker in the case of Kafka)?
>
> If you're referring to the server, then the ensemble recovers from the disk state of the servers. If it is a follower, then there is nothing to recover, the leader already has all the necessary data. If it is the leader, then it will get the ephemeral/session data from disk.
>
> If you're referring to the Kafka broker, then you should be seeing a message like this in your log in the case the broker finds a znode upon registration:
>
> case e: ZkNodeExistsException =>
> throw new RuntimeException("A broker is already registered on the path " + brokerIdPath
> + ". This probably " + "indicates that you either have configured a brokerid that is already in use, or "
> + "else you have shutdown this broker and restarted it faster than the zookeeper "
> + "timeout so it appears to be re-registering.")
>
>
> Does it help?
>
> -Flavio
>
>
>> On 30 Aug 2016, at 14:35, J316 Services <j3...@icloud.com> wrote:
>>
>> Setup is 3 participants + 1 observer.
>>
>> We used 3.5.0 for dynamic configs.
>>
>> What happens when a system forced killed the zookeeper process and it did not have time to clean up ephemeral? Does it recover in that scenario?
>>
>>
>> Thanks.
>>
>> Sent from my iPad
>>
>>> On Aug 30, 2016, at 8:53 AM, Flavio Junqueira <fp...@apache.org> wrote:
>>>
>>> I think you're saying that the session isn't expiring that and the ephemeral isn't getting deleted. Or maybe the session is expiring but the ephemeral isn't being deleted? It'd be great if you could check that the broker session is expiring eventually.
>>>
>>> Since you're on the 3.5 branch, you may want to use 3.5.2-alpha or simply try 3.4.8 as Harsha suggested.
>>>
>>> -Flavio
>>>
>>>> On 29 Aug 2016, at 18:44, Harsha Chintalapani <ka...@harsha.io> wrote:
>>>>
>>>> how many brokers you've in this cluster. Do you try using a stable
>>>> zookeeper release like 3.4.8?
>>>> -Harhsa
>>>>
>>>>> On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
>>>>>
>>>>> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>>>>>
>>>>> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> we would get this occasionally after a weekend reboot/restart.
>>>>>>
>>>>>> we tried restarting a couple of times all to naught.
>>>>>>
>>>>>> we had to delete dk's directory to get his going again.
>>>>>>
>>>>>> any ideas what might cause this issue and suggestions on how to resolve
>>>>>> this?
>>>>>>
>>>>>>
>>>>>> thanks.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Nomar Morado
>>>
>
Re: a broker is already registered on path /brokers/ids/1
Posted by Flavio Junqueira <fp...@apache.org>.
Is the zookeeper process you're referring to the server or the client (broker in the case of Kafka)?
If you're referring to the server, then the ensemble recovers from the disk state of the servers. If it is a follower, then there is nothing to recover, the leader already has all the necessary data. If it is the leader, then it will get the ephemeral/session data from disk.
If you're referring to the Kafka broker, then you should be seeing a message like this in your log in the case the broker finds a znode upon registration:
case e: ZkNodeExistsException =>
throw new RuntimeException("A broker is already registered on the path " + brokerIdPath
+ ". This probably " + "indicates that you either have configured a brokerid that is already in use, or "
+ "else you have shutdown this broker and restarted it faster than the zookeeper "
+ "timeout so it appears to be re-registering.")
Does it help?
-Flavio
> On 30 Aug 2016, at 14:35, J316 Services <j3...@icloud.com> wrote:
>
> Setup is 3 participants + 1 observer.
>
> We used 3.5.0 for dynamic configs.
>
> What happens when a system forced killed the zookeeper process and it did not have time to clean up ephemeral? Does it recover in that scenario?
>
>
> Thanks.
>
> Sent from my iPad
>
>> On Aug 30, 2016, at 8:53 AM, Flavio Junqueira <fp...@apache.org> wrote:
>>
>> I think you're saying that the session isn't expiring that and the ephemeral isn't getting deleted. Or maybe the session is expiring but the ephemeral isn't being deleted? It'd be great if you could check that the broker session is expiring eventually.
>>
>> Since you're on the 3.5 branch, you may want to use 3.5.2-alpha or simply try 3.4.8 as Harsha suggested.
>>
>> -Flavio
>>
>>> On 29 Aug 2016, at 18:44, Harsha Chintalapani <ka...@harsha.io> wrote:
>>>
>>> how many brokers you've in this cluster. Do you try using a stable
>>> zookeeper release like 3.4.8?
>>> -Harhsa
>>>
>>>> On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
>>>>
>>>> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>>>>
>>>> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
>>>> wrote:
>>>>
>>>>> we would get this occasionally after a weekend reboot/restart.
>>>>>
>>>>> we tried restarting a couple of times all to naught.
>>>>>
>>>>> we had to delete dk's directory to get his going again.
>>>>>
>>>>> any ideas what might cause this issue and suggestions on how to resolve
>>>>> this?
>>>>>
>>>>>
>>>>> thanks.
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Nomar Morado
>>
Re: a broker is already registered on path /brokers/ids/1
Posted by Flavio Junqueira <fp...@apache.org>.
Is the zookeeper process you're referring to the server or the client (broker in the case of Kafka)?
If you're referring to the server, then the ensemble recovers from the disk state of the servers. If it is a follower, then there is nothing to recover, the leader already has all the necessary data. If it is the leader, then it will get the ephemeral/session data from disk.
If you're referring to the Kafka broker, then you should be seeing a message like this in your log in the case the broker finds a znode upon registration:
case e: ZkNodeExistsException =>
throw new RuntimeException("A broker is already registered on the path " + brokerIdPath
+ ". This probably " + "indicates that you either have configured a brokerid that is already in use, or "
+ "else you have shutdown this broker and restarted it faster than the zookeeper "
+ "timeout so it appears to be re-registering.")
Does it help?
-Flavio
> On 30 Aug 2016, at 14:35, J316 Services <j3...@icloud.com> wrote:
>
> Setup is 3 participants + 1 observer.
>
> We used 3.5.0 for dynamic configs.
>
> What happens when a system forced killed the zookeeper process and it did not have time to clean up ephemeral? Does it recover in that scenario?
>
>
> Thanks.
>
> Sent from my iPad
>
>> On Aug 30, 2016, at 8:53 AM, Flavio Junqueira <fp...@apache.org> wrote:
>>
>> I think you're saying that the session isn't expiring that and the ephemeral isn't getting deleted. Or maybe the session is expiring but the ephemeral isn't being deleted? It'd be great if you could check that the broker session is expiring eventually.
>>
>> Since you're on the 3.5 branch, you may want to use 3.5.2-alpha or simply try 3.4.8 as Harsha suggested.
>>
>> -Flavio
>>
>>> On 29 Aug 2016, at 18:44, Harsha Chintalapani <ka...@harsha.io> wrote:
>>>
>>> how many brokers you've in this cluster. Do you try using a stable
>>> zookeeper release like 3.4.8?
>>> -Harhsa
>>>
>>>> On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
>>>>
>>>> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>>>>
>>>> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
>>>> wrote:
>>>>
>>>>> we would get this occasionally after a weekend reboot/restart.
>>>>>
>>>>> we tried restarting a couple of times all to naught.
>>>>>
>>>>> we had to delete dk's directory to get his going again.
>>>>>
>>>>> any ideas what might cause this issue and suggestions on how to resolve
>>>>> this?
>>>>>
>>>>>
>>>>> thanks.
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Nomar Morado
>>
Re: a broker is already registered on path /brokers/ids/1
Posted by J316 Services <j3...@icloud.com>.
Setup is 3 participants + 1 observer.
We used 3.5.0 for dynamic configs.
What happens when a system forced killed the zookeeper process and it did not have time to clean up ephemeral? Does it recover in that scenario?
Thanks.
Sent from my iPad
> On Aug 30, 2016, at 8:53 AM, Flavio Junqueira <fp...@apache.org> wrote:
>
> I think you're saying that the session isn't expiring that and the ephemeral isn't getting deleted. Or maybe the session is expiring but the ephemeral isn't being deleted? It'd be great if you could check that the broker session is expiring eventually.
>
> Since you're on the 3.5 branch, you may want to use 3.5.2-alpha or simply try 3.4.8 as Harsha suggested.
>
> -Flavio
>
>> On 29 Aug 2016, at 18:44, Harsha Chintalapani <ka...@harsha.io> wrote:
>>
>> how many brokers you've in this cluster. Do you try using a stable
>> zookeeper release like 3.4.8?
>> -Harhsa
>>
>>> On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
>>>
>>> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>>>
>>> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
>>> wrote:
>>>
>>>> we would get this occasionally after a weekend reboot/restart.
>>>>
>>>> we tried restarting a couple of times all to naught.
>>>>
>>>> we had to delete dk's directory to get his going again.
>>>>
>>>> any ideas what might cause this issue and suggestions on how to resolve
>>>> this?
>>>>
>>>>
>>>> thanks.
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Nomar Morado
>
Re: a broker is already registered on path /brokers/ids/1
Posted by J316 Services <j3...@icloud.com>.
Setup is 3 participants + 1 observer.
We used 3.5.0 for dynamic configs.
What happens when a system forced killed the zookeeper process and it did not have time to clean up ephemeral? Does it recover in that scenario?
Thanks.
Sent from my iPad
> On Aug 30, 2016, at 8:53 AM, Flavio Junqueira <fp...@apache.org> wrote:
>
> I think you're saying that the session isn't expiring that and the ephemeral isn't getting deleted. Or maybe the session is expiring but the ephemeral isn't being deleted? It'd be great if you could check that the broker session is expiring eventually.
>
> Since you're on the 3.5 branch, you may want to use 3.5.2-alpha or simply try 3.4.8 as Harsha suggested.
>
> -Flavio
>
>> On 29 Aug 2016, at 18:44, Harsha Chintalapani <ka...@harsha.io> wrote:
>>
>> how many brokers you've in this cluster. Do you try using a stable
>> zookeeper release like 3.4.8?
>> -Harhsa
>>
>>> On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
>>>
>>> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>>>
>>> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
>>> wrote:
>>>
>>>> we would get this occasionally after a weekend reboot/restart.
>>>>
>>>> we tried restarting a couple of times all to naught.
>>>>
>>>> we had to delete dk's directory to get his going again.
>>>>
>>>> any ideas what might cause this issue and suggestions on how to resolve
>>>> this?
>>>>
>>>>
>>>> thanks.
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Nomar Morado
>
Re: a broker is already registered on path /brokers/ids/1
Posted by Flavio Junqueira <fp...@apache.org>.
I think you're saying that the session isn't expiring that and the ephemeral isn't getting deleted. Or maybe the session is expiring but the ephemeral isn't being deleted? It'd be great if you could check that the broker session is expiring eventually.
Since you're on the 3.5 branch, you may want to use 3.5.2-alpha or simply try 3.4.8 as Harsha suggested.
-Flavio
> On 29 Aug 2016, at 18:44, Harsha Chintalapani <ka...@harsha.io> wrote:
>
> how many brokers you've in this cluster. Do you try using a stable
> zookeeper release like 3.4.8?
> -Harhsa
>
> On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
>
>> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>>
>> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
>> wrote:
>>
>>> we would get this occasionally after a weekend reboot/restart.
>>>
>>> we tried restarting a couple of times all to naught.
>>>
>>> we had to delete dk's directory to get his going again.
>>>
>>> any ideas what might cause this issue and suggestions on how to resolve
>>> this?
>>>
>>>
>>> thanks.
>>>
>>
>>
>>
>> --
>> Regards,
>> Nomar Morado
>>
Re: a broker is already registered on path /brokers/ids/1
Posted by Flavio Junqueira <fp...@apache.org>.
I think you're saying that the session isn't expiring that and the ephemeral isn't getting deleted. Or maybe the session is expiring but the ephemeral isn't being deleted? It'd be great if you could check that the broker session is expiring eventually.
Since you're on the 3.5 branch, you may want to use 3.5.2-alpha or simply try 3.4.8 as Harsha suggested.
-Flavio
> On 29 Aug 2016, at 18:44, Harsha Chintalapani <ka...@harsha.io> wrote:
>
> how many brokers you've in this cluster. Do you try using a stable
> zookeeper release like 3.4.8?
> -Harhsa
>
> On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
>
>> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>>
>> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
>> wrote:
>>
>>> we would get this occasionally after a weekend reboot/restart.
>>>
>>> we tried restarting a couple of times all to naught.
>>>
>>> we had to delete dk's directory to get his going again.
>>>
>>> any ideas what might cause this issue and suggestions on how to resolve
>>> this?
>>>
>>>
>>> thanks.
>>>
>>
>>
>>
>> --
>> Regards,
>> Nomar Morado
>>
Re: a broker is already registered on path /brokers/ids/1
Posted by Harsha Chintalapani <ka...@harsha.io>.
how many brokers you've in this cluster. Do you try using a stable
zookeeper release like 3.4.8?
-Harhsa
On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>
> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
> wrote:
>
> > we would get this occasionally after a weekend reboot/restart.
> >
> > we tried restarting a couple of times all to naught.
> >
> > we had to delete dk's directory to get his going again.
> >
> > any ideas what might cause this issue and suggestions on how to resolve
> > this?
> >
> >
> > thanks.
> >
>
>
>
> --
> Regards,
> Nomar Morado
>
Re: a broker is already registered on path /brokers/ids/1
Posted by Harsha Chintalapani <ka...@harsha.io>.
how many brokers you've in this cluster. Do you try using a stable
zookeeper release like 3.4.8?
-Harhsa
On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado <no...@gmail.com> wrote:
> we are using kafka 0.9.0.1 and zk 3.5.0-alpha
>
> On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
> wrote:
>
> > we would get this occasionally after a weekend reboot/restart.
> >
> > we tried restarting a couple of times all to naught.
> >
> > we had to delete dk's directory to get his going again.
> >
> > any ideas what might cause this issue and suggestions on how to resolve
> > this?
> >
> >
> > thanks.
> >
>
>
>
> --
> Regards,
> Nomar Morado
>
Re: a broker is already registered on path /brokers/ids/1
Posted by Nomar Morado <no...@gmail.com>.
we are using kafka 0.9.0.1 and zk 3.5.0-alpha
On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
wrote:
> we would get this occasionally after a weekend reboot/restart.
>
> we tried restarting a couple of times all to naught.
>
> we had to delete dk's directory to get his going again.
>
> any ideas what might cause this issue and suggestions on how to resolve
> this?
>
>
> thanks.
>
--
Regards,
Nomar Morado
Re: a broker is already registered on path /brokers/ids/1
Posted by Nomar Morado <no...@gmail.com>.
we are using kafka 0.9.0.1 and zk 3.5.0-alpha
On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado <no...@gmail.com>
wrote:
> we would get this occasionally after a weekend reboot/restart.
>
> we tried restarting a couple of times all to naught.
>
> we had to delete dk's directory to get his going again.
>
> any ideas what might cause this issue and suggestions on how to resolve
> this?
>
>
> thanks.
>
--
Regards,
Nomar Morado