KRaft Observer Nodes

[Sanaa Syed <sa...@shopify.com.INVALID>]

Hello,

Currently in Zookeeper, there is the concept of an observer/dummy node that
can be used when scaling up a cluster (
https://zookeeper.apache.org/doc/r3.4.13/zookeeperObservers.html).

Is there an equivalent of an observer node in KRaft mode? Asking because we
use the observer node for our stretched Kafka Clusters (6 zookeeper nodes
across 3 clusters) and would like to find a direct replacement or come up
with an alternative.

Thank you,
Sanaa

Re: KRaft Observer Nodes

[Paolo Patierno <pa...@gmail.com>]

Hi Sanaa,
yes it's enough NOT having the "controller" role to be just an "observer".
As soon as your node has "controller" role configured (in addition to
"broker" or alone) it acts as a "voter" and then being a leader or
"follower" based on the election result.

Thanks
Paolo

On Mon, 25 Mar 2024, 14:25 Sanaa Syed, <sa...@shopify.com.invalid>
wrote:

> Hi Paolo,
>
> If I'm understanding correctly, a kafka broker (aside from the controller
> quorum) can be the observer? Do we need to set the process role to anything
> in this case to indicate that we want a broker to be an observer?
>
> On Sun, Mar 24, 2024 at 3:01 PM Paolo Patierno <pa...@gmail.com>
> wrote:
>
> > Hi Sanaa,
> > in KRaft mode there is the role of "observer" which is typically taken by
> > brokers as part of the data plane. The broker/observer can discover which
> > node is the leader/active controller and fetches the metadata from it (as
> > the "followers" in the KRaft quorum), but cannot vote and cannot become
> > leader.
> > AFAIK it's allowing to change its role at some point if needed to be
> also a
> > controller or anyway taking part at the quorum by also having the
> metadata
> > topic already replicated.
> >
> > Thanks
> > Paolo
> >
> > On Sun, 24 Mar 2024, 19:05 Sanaa Syed, <sa...@shopify.com.invalid>
> > wrote:
> >
> > > Hello,
> > >
> > > Currently in Zookeeper, there is the concept of an observer/dummy node
> > that
> > > can be used when scaling up a cluster (
> > > https://zookeeper.apache.org/doc/r3.4.13/zookeeperObservers.html).
> > >
> > > Is there an equivalent of an observer node in KRaft mode? Asking
> because
> > we
> > > use the observer node for our stretched Kafka Clusters (6 zookeeper
> nodes
> > > across 3 clusters) and would like to find a direct replacement or come
> up
> > > with an alternative.
> > >
> > > Thank you,
> > > Sanaa
> > >
> >
>
>
> --
> Sanaa Syed
>

Re: KRaft Observer Nodes

[Sanaa Syed <sa...@shopify.com.INVALID>]

Hi Paolo,

If I'm understanding correctly, a kafka broker (aside from the controller
quorum) can be the observer? Do we need to set the process role to anything
in this case to indicate that we want a broker to be an observer?

On Sun, Mar 24, 2024 at 3:01 PM Paolo Patierno <pa...@gmail.com>
wrote:

> Hi Sanaa,
> in KRaft mode there is the role of "observer" which is typically taken by
> brokers as part of the data plane. The broker/observer can discover which
> node is the leader/active controller and fetches the metadata from it (as
> the "followers" in the KRaft quorum), but cannot vote and cannot become
> leader.
> AFAIK it's allowing to change its role at some point if needed to be also a
> controller or anyway taking part at the quorum by also having the metadata
> topic already replicated.
>
> Thanks
> Paolo
>
> On Sun, 24 Mar 2024, 19:05 Sanaa Syed, <sa...@shopify.com.invalid>
> wrote:
>
> > Hello,
> >
> > Currently in Zookeeper, there is the concept of an observer/dummy node
> that
> > can be used when scaling up a cluster (
> > https://zookeeper.apache.org/doc/r3.4.13/zookeeperObservers.html).
> >
> > Is there an equivalent of an observer node in KRaft mode? Asking because
> we
> > use the observer node for our stretched Kafka Clusters (6 zookeeper nodes
> > across 3 clusters) and would like to find a direct replacement or come up
> > with an alternative.
> >
> > Thank you,
> > Sanaa
> >
>


-- 
Sanaa Syed

Re: KRaft Observer Nodes

[Paolo Patierno <pa...@gmail.com>]

Hi Sanaa,
in KRaft mode there is the role of "observer" which is typically taken by
brokers as part of the data plane. The broker/observer can discover which
node is the leader/active controller and fetches the metadata from it (as
the "followers" in the KRaft quorum), but cannot vote and cannot become
leader.
AFAIK it's allowing to change its role at some point if needed to be also a
controller or anyway taking part at the quorum by also having the metadata
topic already replicated.

Thanks
Paolo

On Sun, 24 Mar 2024, 19:05 Sanaa Syed, <sa...@shopify.com.invalid>
wrote:

> Hello,
>
> Currently in Zookeeper, there is the concept of an observer/dummy node that
> can be used when scaling up a cluster (
> https://zookeeper.apache.org/doc/r3.4.13/zookeeperObservers.html).
>
> Is there an equivalent of an observer node in KRaft mode? Asking because we
> use the observer node for our stretched Kafka Clusters (6 zookeeper nodes
> across 3 clusters) and would like to find a direct replacement or come up
> with an alternative.
>
> Thank you,
> Sanaa
>