Where is replication factor stored?

[James Cheng <jc...@tivo.com>]

Hi,

Where is the replication factor for a topic stored? It isn't listed at https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper. But the kafka-topics --describe command returns something. Where is it finding that?

Thanks,
-James


________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.

Re: Where is replication factor stored?

[Todd Palino <tp...@gmail.com>]

Sorry, I forgot the tl;dr on that :)

If you want to know the replication factor for a given partition, you want
to check the length of the replica list in the /brokers/topic/(topic) data
for that partition. Note that all the partitions for a topic do not have to
have the same replication factor (you can use partition reassignment to
change it). But if they are not all the same, some of the tooling will
break (such as altering the partition count for the topic).

-Todd


On Fri, Oct 16, 2015 at 5:39 PM, Todd Palino <tp...@gmail.com> wrote:

> Actually, be very careful with this. There are two different things stored
> in Zookeeper, and depending on what you're interested in you want to make
> sure you're looking at the right one.
>
> If you want to know the replica assignment - that is, what brokers a given
> partition is assigned to - you need to look at the following path:
> /brokers/topics/(topic)
>
> The data of that znode is JSON formatted, and the 'partitions' key is a
> dictionary where they key is a string representation of the partition
> number (i.e. it's not 0, it's "0") and the value is a list of the replicas
> that the partition is assigned to. It's worth noting that this replica list
> is also the order in which the preferred leader is selected (the first
> replica in the list that is in sync will be selected as the leader).
>
> If you want to know what the current in sync replicas are - that is, out
> of the assigned replica list, which ones are currently considered to be in
> sync, you need to look at the following path:
> /brokers/topics/(topic)/partitions/(partition number)/state
>
> The data of that znode is also JSON formatted, and the 'isr' key is a list
> of the replicas that are currently considered to be in sync. The important
> distinction here is that this list can be shorter than the actual assigned
> replica list (from the znode above) if not all of the replicas are in sync.
> The state znode also has a 'leader' key which holds the broker ID of the
> replica that is currently the leader for that partition.
>
> -Todd
>
>
> On Fri, Oct 16, 2015 at 5:25 PM, Edward Ribeiro <ed...@gmail.com>
> wrote:
>
>> Hey, Guozhang,
>>
>> On Fri, Oct 16, 2015 at 6:20 PM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>>
>> > The replica list can be from at /brokers/topics/<topic>/
>> > partitions/<partition-id>/state
>> >
>>
>> Nice, good to know. Thanks! :)
>>
>> Regards,
>> Edward​
>>
>
>

Re: Where is replication factor stored?

[Todd Palino <tp...@gmail.com>]

Actually, be very careful with this. There are two different things stored
in Zookeeper, and depending on what you're interested in you want to make
sure you're looking at the right one.

If you want to know the replica assignment - that is, what brokers a given
partition is assigned to - you need to look at the following path:
/brokers/topics/(topic)

The data of that znode is JSON formatted, and the 'partitions' key is a
dictionary where they key is a string representation of the partition
number (i.e. it's not 0, it's "0") and the value is a list of the replicas
that the partition is assigned to. It's worth noting that this replica list
is also the order in which the preferred leader is selected (the first
replica in the list that is in sync will be selected as the leader).

If you want to know what the current in sync replicas are - that is, out of
the assigned replica list, which ones are currently considered to be in
sync, you need to look at the following path:
/brokers/topics/(topic)/partitions/(partition number)/state

The data of that znode is also JSON formatted, and the 'isr' key is a list
of the replicas that are currently considered to be in sync. The important
distinction here is that this list can be shorter than the actual assigned
replica list (from the znode above) if not all of the replicas are in sync.
The state znode also has a 'leader' key which holds the broker ID of the
replica that is currently the leader for that partition.

-Todd


On Fri, Oct 16, 2015 at 5:25 PM, Edward Ribeiro <ed...@gmail.com>
wrote:

> Hey, Guozhang,
>
> On Fri, Oct 16, 2015 at 6:20 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > The replica list can be from at /brokers/topics/<topic>/
> > partitions/<partition-id>/state
> >
>
> Nice, good to know. Thanks! :)
>
> Regards,
> Edward​
>

Re: Where is replication factor stored?

[Edward Ribeiro <ed...@gmail.com>]

Hey, Guozhang,

On Fri, Oct 16, 2015 at 6:20 PM, Guozhang Wang <wa...@gmail.com> wrote:

> The replica list can be from at /brokers/topics/<topic>/
> partitions/<partition-id>/state
>

Nice, good to know. Thanks! :)

Regards,
Edward​

Re: Where is replication factor stored?

[Guozhang Wang <wa...@gmail.com>]

The replica list can be from at /brokers/topics/<topic>/
partitions/<partition-id>/state

Guozhang

On Fri, Oct 16, 2015 at 2:06 PM, Edward Ribeiro <ed...@gmail.com>
wrote:

> Umm... the replica *assignment* gets stored under /brokers/topics/<topic>
> as "partitions" field, no? Under /brokers/topics/<topic>/partitions there
> is a znode for each partition number with a 'state' as a sub-znode, right?
>
> James, by doing via zkCli.sh:
>
> get /brokers/topics/<topic>
>
> You get a json like below:
>
>
> {"version":1,"partitions":{"8":[1],"4":[1],"9":[1],"5":[1],"6":[1],"1":[1],"0":[1],"2":[1],"7":[1],"3":[1]}}
>
> As you can see above, the replica list has only one element ([1]) for each
> partition because ReplicationFactor = 1. You can infer the replication
> factor by getting the *size* of this list for any element.
>
>
> On Fri, Oct 16, 2015 at 5:51 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > We don't store the replication factor per-se. When the topic is created,
> we
> > use the replication factor to generate replica-assignment, and the
> replica
> > assignment gets stored in ZK under:
> /brokers/topics/<topic>/partitions/...
> >
> > This is what gets modified when we re-assign replicas.
> >
> > Hope this helps.
> >
> > Gwen
> >
> > On Fri, Oct 16, 2015 at 12:33 PM, James Cheng <jc...@tivo.com> wrote:
> >
> > > Hi,
> > >
> > > Where is the replication factor for a topic stored? It isn't listed at
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> > .
> > > But the kafka-topics --describe command returns something. Where is it
> > > finding that?
> > >
> > > Thanks,
> > > -James
> > >
> > >
> > > ________________________________
> > >
> > > This email and any attachments may contain confidential and privileged
> > > material for the sole use of the intended recipient. Any review,
> copying,
> > > or distribution of this email (or any attachments) by others is
> > prohibited.
> > > If you are not the intended recipient, please contact the sender
> > > immediately and permanently delete this email and any attachments. No
> > > employee or agent of TiVo Inc. is authorized to conclude any binding
> > > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > > Inc. may only be made by a signed written agreement.
> > >
> >
>



-- 
-- Guozhang

Re: Where is replication factor stored?

[Edward Ribeiro <ed...@gmail.com>]

Umm... the replica *assignment* gets stored under /brokers/topics/<topic>
as "partitions" field, no? Under /brokers/topics/<topic>/partitions there
is a znode for each partition number with a 'state' as a sub-znode, right?

James, by doing via zkCli.sh:

get /brokers/topics/<topic>

You get a json like below:

{"version":1,"partitions":{"8":[1],"4":[1],"9":[1],"5":[1],"6":[1],"1":[1],"0":[1],"2":[1],"7":[1],"3":[1]}}

As you can see above, the replica list has only one element ([1]) for each
partition because ReplicationFactor = 1. You can infer the replication
factor by getting the *size* of this list for any element.


On Fri, Oct 16, 2015 at 5:51 PM, Gwen Shapira <gw...@confluent.io> wrote:

> We don't store the replication factor per-se. When the topic is created, we
> use the replication factor to generate replica-assignment, and the replica
> assignment gets stored in ZK under: /brokers/topics/<topic>/partitions/...
>
> This is what gets modified when we re-assign replicas.
>
> Hope this helps.
>
> Gwen
>
> On Fri, Oct 16, 2015 at 12:33 PM, James Cheng <jc...@tivo.com> wrote:
>
> > Hi,
> >
> > Where is the replication factor for a topic stored? It isn't listed at
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> .
> > But the kafka-topics --describe command returns something. Where is it
> > finding that?
> >
> > Thanks,
> > -James
> >
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>

Re: Where is replication factor stored?

[Guozhang Wang <wa...@gmail.com>]

Gwen is right, I was wrong :P

kafka-topics --describe reads the ZK path that Gwen mentioned to return the
replica list, which gives you the idea about replication factor as well.

Thanks,
Guozhang

On Fri, Oct 16, 2015 at 1:51 PM, Gwen Shapira <gw...@confluent.io> wrote:

> We don't store the replication factor per-se. When the topic is created, we
> use the replication factor to generate replica-assignment, and the replica
> assignment gets stored in ZK under: /brokers/topics/<topic>/partitions/...
>
> This is what gets modified when we re-assign replicas.
>
> Hope this helps.
>
> Gwen
>
> On Fri, Oct 16, 2015 at 12:33 PM, James Cheng <jc...@tivo.com> wrote:
>
> > Hi,
> >
> > Where is the replication factor for a topic stored? It isn't listed at
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> .
> > But the kafka-topics --describe command returns something. Where is it
> > finding that?
> >
> > Thanks,
> > -James
> >
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>



-- 
-- Guozhang

Re: Where is replication factor stored?

[Gwen Shapira <gw...@confluent.io>]

We don't store the replication factor per-se. When the topic is created, we
use the replication factor to generate replica-assignment, and the replica
assignment gets stored in ZK under: /brokers/topics/<topic>/partitions/...

This is what gets modified when we re-assign replicas.

Hope this helps.

Gwen

On Fri, Oct 16, 2015 at 12:33 PM, James Cheng <jc...@tivo.com> wrote:

> Hi,
>
> Where is the replication factor for a topic stored? It isn't listed at
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper.
> But the kafka-topics --describe command returns something. Where is it
> finding that?
>
> Thanks,
> -James
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>

Re: Where is replication factor stored?

[Edward Ribeiro <ed...@gmail.com>]

Umm... Reading the TopicCommand code
https://github.com/apache/kafka/blob/362613347371e9d74184e900ab80ba230940a5c8/core/src/main/scala/kafka/admin/TopicCommand.scala#L192
, it looks like the replication factor (for --describe option, at least) is
calculated by:

1) retrieving the "partitions" map from /brokers/topics/<topic-name>, where
the key is the partition number and the value is a list of replicas and

2) getting the first key-value pair (head) from the map retrieved above,
and from this kv pair then get the size of the replica list (._2.size), no?

When I got the data of get /config/topics/testTopic it came as
{"version":1,"config":{}}, that is, the config was empty (I am running from
a stale -- about two weeks -- trunk).

​What I am missing? :)​

Cheers,
Ed

On Fri, Oct 16, 2015 at 5:19 PM, Guozhang Wang <wa...@gmail.com> wrote:

> Replication factor is stored as topic configs that are introduced since
> 0.8.1, you can find it in the wiki you mentioned.
>
> Guozhang
>
> On Fri, Oct 16, 2015 at 12:33 PM, James Cheng <jc...@tivo.com> wrote:
>
> > Hi,
> >
> > Where is the replication factor for a topic stored? It isn't listed at
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> .
> > But the kafka-topics --describe command returns something. Where is it
> > finding that?
> >
> > Thanks,
> > -James
> >
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>
>
>
> --
> -- Guozhang
>

Re: Where is replication factor stored?

[James Cheng <jc...@tivo.com>]

> On Oct 16, 2015, at 1:19 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> Replication factor is stored as topic configs that are introduced since
> 0.8.1, you can find it in the wiki you mentioned.
>

Ah, I didn't notice the /config section.

But it still doesn't show the replication factor.

[zk: localhost:2181(CONNECTED) 3] get /config/topics/__consumer_offsets
{"version":1,"config":{"segment.bytes":"104857600","cleanup.policy":"compact"}}
cZxid = 0xc0000017a
ctime = Wed Aug 05 22:48:12 UTC 2015
mZxid = 0xc0000017a
mtime = Wed Aug 05 22:48:12 UTC 2015
pZxid = 0xc0000017a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 79
numChildren = 0

I tried that for a number of different topics, and none of them have it.

-James


> Guozhang
>
> On Fri, Oct 16, 2015 at 12:33 PM, James Cheng <jc...@tivo.com> wrote:
>
>> Hi,
>>
>> Where is the replication factor for a topic stored? It isn't listed at
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper.
>> But the kafka-topics --describe command returns something. Where is it
>> finding that?
>>
>> Thanks,
>> -James
>>
>>
>> ________________________________
>>
>> This email and any attachments may contain confidential and privileged
>> material for the sole use of the intended recipient. Any review, copying,
>> or distribution of this email (or any attachments) by others is prohibited.
>> If you are not the intended recipient, please contact the sender
>> immediately and permanently delete this email and any attachments. No
>> employee or agent of TiVo Inc. is authorized to conclude any binding
>> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
>> Inc. may only be made by a signed written agreement.
>>
>
>
>
> --
> -- Guozhang


________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.

Re: Where is replication factor stored?

[Guozhang Wang <wa...@gmail.com>]

Replication factor is stored as topic configs that are introduced since
0.8.1, you can find it in the wiki you mentioned.

Guozhang

On Fri, Oct 16, 2015 at 12:33 PM, James Cheng <jc...@tivo.com> wrote:

> Hi,
>
> Where is the replication factor for a topic stored? It isn't listed at
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper.
> But the kafka-topics --describe command returns something. Where is it
> finding that?
>
> Thanks,
> -James
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>



-- 
-- Guozhang