You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by kant kodali <ka...@gmail.com> on 2016/09/17 08:36:44 UTC
can one topic be registered in multiple brokers?
can one topic be registered in multiple brokers? if so, which component of
kafka decides which broker should get the message for that particular topic?
Thanks!
Re: can one topic be registered in multiple brokers?
Posted by Radoslaw Gruchalski <ra...@gruchalski.com>.
Kafka uses murmur2 key hashing by default. You can also create your own custom partitioner. The partitioner can be set on per producer / consumer basis.
--
Best regards,
Rad
On Sat, Sep 17, 2016 at 11:01 AM +0200, "kant kodali" <ka...@gmail.com> wrote:
so Zookeeper will select which broker it should direct the message to if I have
3 brokers and 3 partitions of a topic?
I only finished the benchmarks of Kafka and NSQ and still working NATS Streaming
Server (one more day I will finish it). But so far Kafka had a great throughput
with Java Client about 55000 messages per second(receive throughput) and 70K
messages per second(sendthrougput)each message size is 1KB. so yeah something
wrong with node.js client
NSQ was ~ 3K receive throughput and~4K send throughput
one thing about my NSQ benchmark is first of all I am new to Go and I have the
following code (pasting only the lines that are relevant). I just dont know if
w.Publish is a blocking call? If it is a blocking call I have to see if the
library offers any non-blocking async call since the java version I wrote for
Kafka was async. That way I am comparing apples to apples
w, _ := nsq.NewProducer("172.31.18.175:4150", config) for i := 0; i<500000; i++
{ err := w.Publish("nsq_go_test", []byte(data)) if err != nil {
log.Panic("Could not connect") } }
From what I heard beating NATS is going to be harder but I rather run my own
experiment than going with what I heard. Also it
On Sat, Sep 17, 2016 1:38 AM, Ali Akhtar ali.rac200@gmail.com
wrote:
You can create multiple partitions of a topic and kafka will attempt to
distribute them evenly.
E.g if you have 3 brokers and you create 3 partitions of a topic, each
broker will be the leader of 1 of the 3 partitions.
P.S how did the benchmarking go?
On Sat, Sep 17, 2016 at 1:36 PM, kant kodali wrote:
> can one topic be registered in multiple brokers? if so, which component of
> kafka decides which broker should get the message for that particular
> topic?
> Thanks!
Re: can one topic be registered in multiple brokers?
Posted by kant kodali <ka...@gmail.com>.
so Zookeeper will select which broker it should direct the message to if I have
3 brokers and 3 partitions of a topic?
I only finished the benchmarks of Kafka and NSQ and still working NATS Streaming
Server (one more day I will finish it). But so far Kafka had a great throughput
with Java Client about 55000 messages per second(receive throughput) and 70K
messages per second(sendthrougput)each message size is 1KB. so yeah something
wrong with node.js client
NSQ was ~ 3K receive throughput and~4K send throughput
one thing about my NSQ benchmark is first of all I am new to Go and I have the
following code (pasting only the lines that are relevant). I just dont know if
w.Publish is a blocking call? If it is a blocking call I have to see if the
library offers any non-blocking async call since the java version I wrote for
Kafka was async. That way I am comparing apples to apples
w, _ := nsq.NewProducer("172.31.18.175:4150", config) for i := 0; i<500000; i++
{ err := w.Publish("nsq_go_test", []byte(data)) if err != nil {
log.Panic("Could not connect") } }
From what I heard beating NATS is going to be harder but I rather run my own
experiment than going with what I heard. Also it
On Sat, Sep 17, 2016 1:38 AM, Ali Akhtar ali.rac200@gmail.com
wrote:
You can create multiple partitions of a topic and kafka will attempt to
distribute them evenly.
E.g if you have 3 brokers and you create 3 partitions of a topic, each
broker will be the leader of 1 of the 3 partitions.
P.S how did the benchmarking go?
On Sat, Sep 17, 2016 at 1:36 PM, kant kodali <ka...@gmail.com> wrote:
> can one topic be registered in multiple brokers? if so, which component of
> kafka decides which broker should get the message for that particular
> topic?
> Thanks!
Re: can one topic be registered in multiple brokers?
Posted by Ali Akhtar <al...@gmail.com>.
You can create multiple partitions of a topic and kafka will attempt to
distribute them evenly.
E.g if you have 3 brokers and you create 3 partitions of a topic, each
broker will be the leader of 1 of the 3 partitions.
P.S how did the benchmarking go?
On Sat, Sep 17, 2016 at 1:36 PM, kant kodali <ka...@gmail.com> wrote:
> can one topic be registered in multiple brokers? if so, which component of
> kafka decides which broker should get the message for that particular
> topic?
> Thanks!