You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Lung, Paul" <pl...@ebay.com> on 2014/06/25 03:20:35 UTC
Blacklisting Brokers
Hi All,
Is there anyway to blacklist brokers? Sometimes we run into situations where there are certain hardware failures on a broker machine, and the machines goes into a “half dead” state. The broker process is up and participating in the cluster, but it can’t actually transmit messages properly. Sometimes, even SSH fails on the machines, and we can’t log in to shutdown the broker process. The only way to resolve this issue at this point is to shutdown the machine, which is not always easy in a large data center scenario. The consequences are disastrous if this half dead broker is elected as the leader. So in this situation, it would be nice to be able to quickly black list a broker.
Thank you.
Paul Lung
Re: Blacklisting Brokers
Posted by Neha Narkhede <ne...@gmail.com>.
Could you file a JIRA for this? We may have to think through this
carefully. Black listing should be done in a way that maintains the
replication factor of the partition. Un blacklisting would then either
increase the replication factor or require carefully removing the replicas
on that broker.
Thanks,
Neha
On Thu, Jun 26, 2014 at 9:27 AM, Lung, Paul <pl...@ebay.com> wrote:
> Yes I have seen it elected as the leader. The broker process is still up,
> but due to some unknown hardware failure, it¹s not working 100%. It is,
> however, able to participate in the cluster, and got itself elected as the
> leader. I¹m not sure what the hardware problem is though, since I couldn¹t
> log into the machine to check.
>
> Paul
>
> On 6/26/14, 9:03 AM, "Neha Narkhede" <ne...@gmail.com> wrote:
>
> >If ssh fails on the broker machine, I'm not why the broker would be in a
> >state where it maintains it's registration in zookeeper? If so, it will
> >automatically be marked dead and will not get elected as the leader. Have
> >you seen a case where such a broker became the leader? Could you elaborate
> >more on the type of failure that you describe?
> >
> >
> >
> >
> >On Tue, Jun 24, 2014 at 6:20 PM, Lung, Paul <pl...@ebay.com> wrote:
> >
> >> Hi All,
> >>
> >> Is there anyway to blacklist brokers? Sometimes we run into situations
> >> where there are certain hardware failures on a broker machine, and the
> >> machines goes into a ³half dead² state. The broker process is up and
> >> participating in the cluster, but it can¹t actually transmit messages
> >> properly. Sometimes, even SSH fails on the machines, and we can¹t log
> >>in to
> >> shutdown the broker process. The only way to resolve this issue at this
> >> point is to shutdown the machine, which is not always easy in a large
> >>data
> >> center scenario. The consequences are disastrous if this half dead
> >>broker
> >> is elected as the leader. So in this situation, it would be nice to be
> >>able
> >> to quickly black list a broker.
> >>
> >> Thank you.
> >> Paul Lung
> >>
>
>
Re: Blacklisting Brokers
Posted by "Lung, Paul" <pl...@ebay.com>.
Yes I have seen it elected as the leader. The broker process is still up,
but due to some unknown hardware failure, it¹s not working 100%. It is,
however, able to participate in the cluster, and got itself elected as the
leader. I¹m not sure what the hardware problem is though, since I couldn¹t
log into the machine to check.
Paul
On 6/26/14, 9:03 AM, "Neha Narkhede" <ne...@gmail.com> wrote:
>If ssh fails on the broker machine, I'm not why the broker would be in a
>state where it maintains it's registration in zookeeper? If so, it will
>automatically be marked dead and will not get elected as the leader. Have
>you seen a case where such a broker became the leader? Could you elaborate
>more on the type of failure that you describe?
>
>
>
>
>On Tue, Jun 24, 2014 at 6:20 PM, Lung, Paul <pl...@ebay.com> wrote:
>
>> Hi All,
>>
>> Is there anyway to blacklist brokers? Sometimes we run into situations
>> where there are certain hardware failures on a broker machine, and the
>> machines goes into a ³half dead² state. The broker process is up and
>> participating in the cluster, but it can¹t actually transmit messages
>> properly. Sometimes, even SSH fails on the machines, and we can¹t log
>>in to
>> shutdown the broker process. The only way to resolve this issue at this
>> point is to shutdown the machine, which is not always easy in a large
>>data
>> center scenario. The consequences are disastrous if this half dead
>>broker
>> is elected as the leader. So in this situation, it would be nice to be
>>able
>> to quickly black list a broker.
>>
>> Thank you.
>> Paul Lung
>>
Re: Blacklisting Brokers
Posted by Neha Narkhede <ne...@gmail.com>.
If ssh fails on the broker machine, I'm not why the broker would be in a
state where it maintains it's registration in zookeeper? If so, it will
automatically be marked dead and will not get elected as the leader. Have
you seen a case where such a broker became the leader? Could you elaborate
more on the type of failure that you describe?
On Tue, Jun 24, 2014 at 6:20 PM, Lung, Paul <pl...@ebay.com> wrote:
> Hi All,
>
> Is there anyway to blacklist brokers? Sometimes we run into situations
> where there are certain hardware failures on a broker machine, and the
> machines goes into a “half dead” state. The broker process is up and
> participating in the cluster, but it can’t actually transmit messages
> properly. Sometimes, even SSH fails on the machines, and we can’t log in to
> shutdown the broker process. The only way to resolve this issue at this
> point is to shutdown the machine, which is not always easy in a large data
> center scenario. The consequences are disastrous if this half dead broker
> is elected as the leader. So in this situation, it would be nice to be able
> to quickly black list a broker.
>
> Thank you.
> Paul Lung
>