Newly added partitions in 0.8.1.1 dont see data immediately

[Prakash Gowri Shankor <pr...@gmail.com>]

Hi,

I used the add partition functionality in create-topics to alter a previous
topic and increase the partitions. I noticed that after the new partitions
were added, they dont receive data immediately from the producer unless a
new producer is started up or the old producer is restarted.
Here is the sequence of steps:
1) start cluster with 2 brokers.
2) create topic with 2 partitions and replication factor 2.
3) start 4 consumers in a group. 2 consumers are redundant.
4) Start the performance producer to send messages.
5) Now alter the topic to have 4 partitions.
6) zookeeper immediately shows the new partitions
7) On watching the partitions through Kafka Offset Monitor, I dont see any
messages going into the 2 new partitions.
8) However, if i start a new performance producer or restart the one in 4)
I see that all the partitions are written to.

Why does 7) happen ? Is this again related to the sticky partitioning
behavior change in 0.8.11 ?
My producer.properties has topic.metadata.refresh.interval.ms=1000

Thanks,
Prakash

Re: Newly added partitions in 0.8.1.1 dont see data immediately

[Jun Rao <ju...@gmail.com>]

With  topic.metadata.refresh.interval.ms=1000, the producer should refresh
metadata and pick up the new partitions after 1 sec. Do you see metadata
being refreshed? You may have to turn on the debug level logging.

Thanks,

Jun


On Mon, Jun 16, 2014 at 3:18 PM, Prakash Gowri Shankor <
prakash.shankor@gmail.com> wrote:

> Hi,
>
> I used the add partition functionality in create-topics to alter a previous
> topic and increase the partitions. I noticed that after the new partitions
> were added, they dont receive data immediately from the producer unless a
> new producer is started up or the old producer is restarted.
> Here is the sequence of steps:
> 1) start cluster with 2 brokers.
> 2) create topic with 2 partitions and replication factor 2.
> 3) start 4 consumers in a group. 2 consumers are redundant.
> 4) Start the performance producer to send messages.
> 5) Now alter the topic to have 4 partitions.
> 6) zookeeper immediately shows the new partitions
> 7) On watching the partitions through Kafka Offset Monitor, I dont see any
> messages going into the 2 new partitions.
> 8) However, if i start a new performance producer or restart the one in 4)
> I see that all the partitions are written to.
>
> Why does 7) happen ? Is this again related to the sticky partitioning
> behavior change in 0.8.11 ?
> My producer.properties has topic.metadata.refresh.interval.ms=1000
>
> Thanks,
> Prakash
>