You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jason Huang <ja...@icare.com> on 2013/01/07 17:49:22 UTC
Kafka 0.8 - KeyedMessage?
Hello,
I did some search on the web but couldn't find any documentation for
0.8 so I am trying to ask here:
KeyedMessage is introduced in 0.8.0:
class KeyedMessage[K, V](val topic: String, val key: K, val message: V)
Does the parameter "key" = "partition key"?
If I build a KeyedMessage with a specific key string, then the message
will be stored to this partition, right?
If I build a KeyedMessage without a key, then the message will be
distributed to a random partition belonging to this topic? Or will it
always use a certain default partition?
thanks,
Jason
Re: Kafka 0.8 - KeyedMessage?
Posted by Jason Huang <ja...@icare.com>.
I see.
This makes sense.
thanks Neha,
Jason
On Mon, Jan 7, 2013 at 1:52 PM, Neha Narkhede <ne...@gmail.com> wrote:
> Jason,
>
> If you specify a key for a message but do not explicitly wire in a
> partitioner, messages with the same key will still land up in the same
> partition. This is because we use a default partitioner that does a simple
> hash(key) % num_partitions.
>
> Thanks,
> Neha
>
>
> On Mon, Jan 7, 2013 at 9:30 AM, Jason Huang <ja...@icare.com> wrote:
>
>> Jun,
>>
>> Thanks for the response. If I understand you correctly, messages with
>> the same key will not be automatically stored at the same partition
>> unless I implement a partition function to route the message based on
>> the key?
>>
>> The quick start guide for 0.7 has the following:
>> "Send a message with a partition key. Messages with the same key are
>> sent to the same partition
>> ProducerData<String, String> data = new ProducerData<String,
>> String>("test-topic", "test-key", "test-message");
>> producer.send(data);"
>>
>> So... there is no simple way to send messages to a certain default
>> partition in 0.8?
>>
>> thanks,
>>
>> Jason
>>
>>
>> On Mon, Jan 7, 2013 at 12:04 PM, Jun Rao <ju...@gmail.com> wrote:
>> > Jason,
>> >
>> > In 0.8, each message can optionally have a key. The key is retained as
>> part
>> > of the message and will be stored in the broker. One can design a
>> partition
>> > function to route the message based on the key. The default partitioner
>> > ignores the key and selects a partition at random.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Mon, Jan 7, 2013 at 8:49 AM, Jason Huang <ja...@icare.com>
>> wrote:
>> >
>> >> Hello,
>> >>
>> >> I did some search on the web but couldn't find any documentation for
>> >> 0.8 so I am trying to ask here:
>> >>
>> >> KeyedMessage is introduced in 0.8.0:
>> >> class KeyedMessage[K, V](val topic: String, val key: K, val message: V)
>> >>
>> >> Does the parameter "key" = "partition key"?
>> >>
>> >> If I build a KeyedMessage with a specific key string, then the message
>> >> will be stored to this partition, right?
>> >>
>> >> If I build a KeyedMessage without a key, then the message will be
>> >> distributed to a random partition belonging to this topic? Or will it
>> >> always use a certain default partition?
>> >>
>> >> thanks,
>> >>
>> >> Jason
>> >>
>>
Re: Kafka 0.8 - KeyedMessage?
Posted by Neha Narkhede <ne...@gmail.com>.
Jason,
If you specify a key for a message but do not explicitly wire in a
partitioner, messages with the same key will still land up in the same
partition. This is because we use a default partitioner that does a simple
hash(key) % num_partitions.
Thanks,
Neha
On Mon, Jan 7, 2013 at 9:30 AM, Jason Huang <ja...@icare.com> wrote:
> Jun,
>
> Thanks for the response. If I understand you correctly, messages with
> the same key will not be automatically stored at the same partition
> unless I implement a partition function to route the message based on
> the key?
>
> The quick start guide for 0.7 has the following:
> "Send a message with a partition key. Messages with the same key are
> sent to the same partition
> ProducerData<String, String> data = new ProducerData<String,
> String>("test-topic", "test-key", "test-message");
> producer.send(data);"
>
> So... there is no simple way to send messages to a certain default
> partition in 0.8?
>
> thanks,
>
> Jason
>
>
> On Mon, Jan 7, 2013 at 12:04 PM, Jun Rao <ju...@gmail.com> wrote:
> > Jason,
> >
> > In 0.8, each message can optionally have a key. The key is retained as
> part
> > of the message and will be stored in the broker. One can design a
> partition
> > function to route the message based on the key. The default partitioner
> > ignores the key and selects a partition at random.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 7, 2013 at 8:49 AM, Jason Huang <ja...@icare.com>
> wrote:
> >
> >> Hello,
> >>
> >> I did some search on the web but couldn't find any documentation for
> >> 0.8 so I am trying to ask here:
> >>
> >> KeyedMessage is introduced in 0.8.0:
> >> class KeyedMessage[K, V](val topic: String, val key: K, val message: V)
> >>
> >> Does the parameter "key" = "partition key"?
> >>
> >> If I build a KeyedMessage with a specific key string, then the message
> >> will be stored to this partition, right?
> >>
> >> If I build a KeyedMessage without a key, then the message will be
> >> distributed to a random partition belonging to this topic? Or will it
> >> always use a certain default partition?
> >>
> >> thanks,
> >>
> >> Jason
> >>
>
Re: Kafka 0.8 - KeyedMessage?
Posted by Jason Huang <ja...@icare.com>.
Jun,
Thanks for the response. If I understand you correctly, messages with
the same key will not be automatically stored at the same partition
unless I implement a partition function to route the message based on
the key?
The quick start guide for 0.7 has the following:
"Send a message with a partition key. Messages with the same key are
sent to the same partition
ProducerData<String, String> data = new ProducerData<String,
String>("test-topic", "test-key", "test-message");
producer.send(data);"
So... there is no simple way to send messages to a certain default
partition in 0.8?
thanks,
Jason
On Mon, Jan 7, 2013 at 12:04 PM, Jun Rao <ju...@gmail.com> wrote:
> Jason,
>
> In 0.8, each message can optionally have a key. The key is retained as part
> of the message and will be stored in the broker. One can design a partition
> function to route the message based on the key. The default partitioner
> ignores the key and selects a partition at random.
>
> Thanks,
>
> Jun
>
> On Mon, Jan 7, 2013 at 8:49 AM, Jason Huang <ja...@icare.com> wrote:
>
>> Hello,
>>
>> I did some search on the web but couldn't find any documentation for
>> 0.8 so I am trying to ask here:
>>
>> KeyedMessage is introduced in 0.8.0:
>> class KeyedMessage[K, V](val topic: String, val key: K, val message: V)
>>
>> Does the parameter "key" = "partition key"?
>>
>> If I build a KeyedMessage with a specific key string, then the message
>> will be stored to this partition, right?
>>
>> If I build a KeyedMessage without a key, then the message will be
>> distributed to a random partition belonging to this topic? Or will it
>> always use a certain default partition?
>>
>> thanks,
>>
>> Jason
>>
Re: Kafka 0.8 - KeyedMessage?
Posted by Jun Rao <ju...@gmail.com>.
Jason,
In 0.8, each message can optionally have a key. The key is retained as part
of the message and will be stored in the broker. One can design a partition
function to route the message based on the key. The default partitioner
ignores the key and selects a partition at random.
Thanks,
Jun
On Mon, Jan 7, 2013 at 8:49 AM, Jason Huang <ja...@icare.com> wrote:
> Hello,
>
> I did some search on the web but couldn't find any documentation for
> 0.8 so I am trying to ask here:
>
> KeyedMessage is introduced in 0.8.0:
> class KeyedMessage[K, V](val topic: String, val key: K, val message: V)
>
> Does the parameter "key" = "partition key"?
>
> If I build a KeyedMessage with a specific key string, then the message
> will be stored to this partition, right?
>
> If I build a KeyedMessage without a key, then the message will be
> distributed to a random partition belonging to this topic? Or will it
> always use a certain default partition?
>
> thanks,
>
> Jason
>