You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by tao xiao <xi...@gmail.com> on 2015/03/09 12:10:26 UTC
Topics are not evenly distributed to streams using Range partition assignment
Hi,
I created a message stream in my consumer using connector
.createMessageStreamsByFilter(new Whitelist("mm-benchmark-test\\w*"), 5); I
have 5 topics in my cluster and each of the topic has only one partition.
My understanding of wildcard stream is that multiple streams are shared
between selected topics. In my case 5 streams should be shared between 5
different topics. But when I looked at the log it showed a different story
2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
owned partition 0 for topic mm-benchmark-test2
2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
owned partition 0 for topic mm-benchmark-test
2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
owned partition 0 for topic mm-benchmark-test1
2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
owned partition 0 for topic mm-benchmark-test4
2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
owned partition 0 for topic mm-benchmark-test3
As indicated from the log only one stream was assigned to all topics. I
just wanted to know if this is expected behavior? if yes how do we evenly
distribute topics across different streams? by using roundrobin assigner?
--
Regards,
Tao
Re: Topics are not evenly distributed to streams using Range
partition assignment
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Hi Tao,
That is expected behavior. You can use set
partition.assignment.strategy=roundrobin in consumer config. It will take
all the partitions from all topics and do a round robin assignment,
whereas range only take partitions for each individual topic for
assignment.
Jiangjie (Becket) Qin
On 3/9/15, 4:10 AM, "tao xiao" <xi...@gmail.com> wrote:
>Hi,
>
>I created a message stream in my consumer using connector
>.createMessageStreamsByFilter(new Whitelist("mm-benchmark-test\\w*"), 5);
>I
>have 5 topics in my cluster and each of the topic has only one partition.
>My understanding of wildcard stream is that multiple streams are shared
>between selected topics. In my case 5 streams should be shared between 5
>different topics. But when I looked at the log it showed a different story
>
>2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
>[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
>test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
>owned partition 0 for topic mm-benchmark-test2
>
>2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
>[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
>test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
>owned partition 0 for topic mm-benchmark-test
>
>2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
>[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
>test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
>owned partition 0 for topic mm-benchmark-test1
>
>2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
>[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
>test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
>owned partition 0 for topic mm-benchmark-test4
>
>2015-03-09 19:02:36 INFO kafka.utils.Logging$class:68 -
>[test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75],
>test12345667ffff_LM-SHC-00950667-1425898953590-d99e2d75-0 successfully
>owned partition 0 for topic mm-benchmark-test3
>
>
>As indicated from the log only one stream was assigned to all topics. I
>just wanted to know if this is expected behavior? if yes how do we evenly
>distribute topics across different streams? by using roundrobin assigner?
>
>
>--
>Regards,
>Tao