You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jörn Franke <jo...@gmail.com> on 2019/08/27 05:32:54 UTC
Re: Byzantine Fault Tolerance Implementation
What kind of stability problems do you have? It is surprising to me that you have them and it is unlikely that you have them due to a specific consensus algorithm. If you have stability issues then I would look at your architecture for weak spots.
Btw. Paxos is a consensus mechanism. Bft just describes a specific type of failures in distributed systems, so implement BFT does probably not make sense.
> Am 27.08.2019 um 06:36 schrieb Nayak, Soumya R. <sn...@firstam.com>:
>
> Hi Team,
>
> Currently Zookeeper and Kafka cluster are Crash Fault Tolerant.
> Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is there any plan in future or current in progress where zookeeper will be implemented with a BFT algorithm. This might help to have a more stable distributed environment when we have the cluster across different machines.
>
> Regards,
> Soumya
>
> ******************************************************************************************
> This message may contain confidential or proprietary information intended only for the use of the
> addressee(s) named above or may contain information that is legally privileged. If you are
> not the intended addressee, or the person responsible for delivering it to the intended addressee,
> you are hereby notified that reading, disseminating, distributing or copying this message is strictly
> prohibited. If you have received this message by mistake, please immediately notify us by
> replying to the message and delete the original message and any copies immediately thereafter.
>
> If you received this email as a commercial message and would like to opt out of future commercial
> messages, please let us know and we will remove you from our distribution list.
>
> Thank you.~
> ******************************************************************************************
> FAFLD
Byzantine Fault Tolerance Implementation
Posted by "Nayak, Soumya R." <sn...@firstam.com>.
Thanks Boyang.
Will check that link and update there.
Regards,
Soumya
-----Original Message-----
From: Boyang Chen <re...@gmail.com>
Sent: Tuesday, August 27, 2019 11:50 AM
To: users@kafka.apache.org
Subject: Re: Byzantine Fault Tolerance Implementation
Hey Nayak,
there is an on-going KIP in the community about deprecating zookeeper:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
It should be a good place to raise your question about making consensus algorithm pluggable in the future.
Boyang
On Mon, Aug 26, 2019 at 10:41 PM Nayak, Soumya R. <sn...@firstam.com>
wrote:
> Hi Jorn,
>
> I was talking with the context of Hyperledger Fabric Blockchain where
> the cluster of kafka zookeeper is used where there might be multiple
> orgs taking part in the network and transactions where a single system
> getting failed or a malicious node might disrupt the whole network
> which would cost a lot.
> So on that aspect I was asking if its possible to have a pluggable
> algorithm for zookeeper ?
>
> Regards,
> Soumya
> -----Original Message-----
> From: Jörn Franke <jo...@gmail.com>
> Sent: Tuesday, August 27, 2019 11:03 AM
> To: users@kafka.apache.org
> Subject: Re: Byzantine Fault Tolerance Implementation
>
> What kind of stability problems do you have? It is surprising to me
> that you have them and it is unlikely that you have them due to a
> specific consensus algorithm. If you have stability issues then I
> would look at your architecture for weak spots.
>
> Btw. Paxos is a consensus mechanism. Bft just describes a specific
> type of failures in distributed systems, so implement BFT does
> probably not make sense.
>
> > Am 27.08.2019 um 06:36 schrieb Nayak, Soumya R. <sn...@firstam.com>:
> >
> > Hi Team,
> >
> > Currently Zookeeper and Kafka cluster are Crash Fault Tolerant.
> > Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is
> > there
> any plan in future or current in progress where zookeeper will be
> implemented with a BFT algorithm. This might help to have a more
> stable distributed environment when we have the cluster across different machines.
> >
> > Regards,
> > Soumya
> >
> > ********************************************************************
> > **
> > ******************** This message may contain confidential or
> > proprietary information intended only for the use of the
> > addressee(s) named above or may contain information that is legally
> > privileged. If you are not the intended addressee, or the person
> > responsible for delivering it to the intended addressee, you are
> > hereby notified that reading, disseminating, distributing or copying
> > this message is strictly prohibited. If you have received this
> > message
> by mistake, please immediately notify us by replying to the message
> and delete the original message and any copies immediately thereafter.
> >
> > If you received this email as a commercial message and would like to
> > opt out of future commercial messages, please let us know and we
> > will
> remove you from our distribution list.
> >
> > Thank you.~
> > ********************************************************************
> > **
> > ********************
> > FAFLD
>
Re: Byzantine Fault Tolerance Implementation
Posted by Boyang Chen <re...@gmail.com>.
Hey Nayak,
there is an on-going KIP in the community about deprecating zookeeper:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
It should be a good place to raise your question about making consensus
algorithm pluggable in the future.
Boyang
On Mon, Aug 26, 2019 at 10:41 PM Nayak, Soumya R. <sn...@firstam.com>
wrote:
> Hi Jorn,
>
> I was talking with the context of Hyperledger Fabric Blockchain where the
> cluster of kafka zookeeper is used where there might be multiple orgs
> taking part in the network and transactions where a single system getting
> failed or a malicious node might disrupt the whole network which would cost
> a lot.
> So on that aspect I was asking if its possible to have a pluggable
> algorithm for zookeeper ?
>
> Regards,
> Soumya
> -----Original Message-----
> From: Jörn Franke <jo...@gmail.com>
> Sent: Tuesday, August 27, 2019 11:03 AM
> To: users@kafka.apache.org
> Subject: Re: Byzantine Fault Tolerance Implementation
>
> What kind of stability problems do you have? It is surprising to me that
> you have them and it is unlikely that you have them due to a specific
> consensus algorithm. If you have stability issues then I would look at your
> architecture for weak spots.
>
> Btw. Paxos is a consensus mechanism. Bft just describes a specific type of
> failures in distributed systems, so implement BFT does probably not make
> sense.
>
> > Am 27.08.2019 um 06:36 schrieb Nayak, Soumya R. <sn...@firstam.com>:
> >
> > Hi Team,
> >
> > Currently Zookeeper and Kafka cluster are Crash Fault Tolerant.
> > Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is there
> any plan in future or current in progress where zookeeper will be
> implemented with a BFT algorithm. This might help to have a more stable
> distributed environment when we have the cluster across different machines.
> >
> > Regards,
> > Soumya
> >
> > **********************************************************************
> > ******************** This message may contain confidential or
> > proprietary information intended only for the use of the
> > addressee(s) named above or may contain information that is legally
> > privileged. If you are not the intended addressee, or the person
> > responsible for delivering it to the intended addressee, you are
> > hereby notified that reading, disseminating, distributing or copying
> > this message is strictly prohibited. If you have received this message
> by mistake, please immediately notify us by replying to the message and
> delete the original message and any copies immediately thereafter.
> >
> > If you received this email as a commercial message and would like to
> > opt out of future commercial messages, please let us know and we will
> remove you from our distribution list.
> >
> > Thank you.~
> > **********************************************************************
> > ********************
> > FAFLD
>
Byzantine Fault Tolerance Implementation
Posted by "Nayak, Soumya R." <sn...@firstam.com>.
Hi Jorn,
I was talking with the context of Hyperledger Fabric Blockchain where the cluster of kafka zookeeper is used where there might be multiple orgs taking part in the network and transactions where a single system getting failed or a malicious node might disrupt the whole network which would cost a lot.
So on that aspect I was asking if its possible to have a pluggable algorithm for zookeeper ?
Regards,
Soumya
-----Original Message-----
From: Jörn Franke <jo...@gmail.com>
Sent: Tuesday, August 27, 2019 11:03 AM
To: users@kafka.apache.org
Subject: Re: Byzantine Fault Tolerance Implementation
What kind of stability problems do you have? It is surprising to me that you have them and it is unlikely that you have them due to a specific consensus algorithm. If you have stability issues then I would look at your architecture for weak spots.
Btw. Paxos is a consensus mechanism. Bft just describes a specific type of failures in distributed systems, so implement BFT does probably not make sense.
> Am 27.08.2019 um 06:36 schrieb Nayak, Soumya R. <sn...@firstam.com>:
>
> Hi Team,
>
> Currently Zookeeper and Kafka cluster are Crash Fault Tolerant.
> Zookeeper uses a version of Paxos - Zookeeper atomic broadcast. Is there any plan in future or current in progress where zookeeper will be implemented with a BFT algorithm. This might help to have a more stable distributed environment when we have the cluster across different machines.
>
> Regards,
> Soumya
>
> **********************************************************************
> ******************** This message may contain confidential or
> proprietary information intended only for the use of the
> addressee(s) named above or may contain information that is legally
> privileged. If you are not the intended addressee, or the person
> responsible for delivering it to the intended addressee, you are
> hereby notified that reading, disseminating, distributing or copying
> this message is strictly prohibited. If you have received this message by mistake, please immediately notify us by replying to the message and delete the original message and any copies immediately thereafter.
>
> If you received this email as a commercial message and would like to
> opt out of future commercial messages, please let us know and we will remove you from our distribution list.
>
> Thank you.~
> **********************************************************************
> ********************
> FAFLD