You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Gaurav Agarwal <ga...@gmail.com> on 2015/02/26 19:53:25 UTC
kafka partitions api
After retrieving a kafka stream or kafka message how to get the
corresponding partition number to which it belongs ? I am using kafka
version 0.8.1.
More specifically kafka.consumer.KafkaStream and
kafka.message.MessageAndMetaData classes, does not provide API to retrieve
partition number. Are there any other API's to get the partition number?
IF there are multiple partitions of a topic ,Do i need to declare from java
code how many partitions the topic contains or i can leave it topic
Kafkastream will take the partition information from kafka broker at
runtime.?
Re: kafka partitions api
Posted by tao xiao <xi...@gmail.com>.
Alex,
You can get partition from MessageAndMetadata as partition is exported via
constructor parameter
On Fri, Feb 27, 2015 at 2:12 PM, Alex Melville <am...@g.hmc.edu> wrote:
> Tao and Gaurav,
>
>
> After looking through the source code in Kafka v8.2.0, I don't see any
> partition() function on the MessageAndMetadata object. Here's the class's
> source:
>
>
> package kafka.message
>
> import kafka.serializer.Decoder
> import kafka.utils.Utils
>
> case class MessageAndMetadata[K, V](topic: String, partition: Int,
> private val rawMessage: Message,
> offset: Long,
> keyDecoder: Decoder[K],
> valueDecoder: Decoder[V]) {
>
> /**
> * Return the decoded message key and payload
> */
> def key(): K = if(rawMessage.key == null) null.asInstanceOf[K] else
> keyDecoder.fromBytes(Utils.readBytes(rawMessage.key))
>
> def message(): V = if(rawMessage.isNull) null.asInstanceOf[V] else
> valueDecoder.fromBytes(Utils.readBytes(rawMessage.payload))
>
>
>
> -Alex M.
>
>
> On Thu, Feb 26, 2015 at 9:54 PM, Gaurav Agarwal <ga...@gmail.com>
> wrote:
>
> > that's fine to me , you can open a separate thread , But the original
> > question when the consumerconnector got connected to a separate topic ,
> > Whether KafkaStream will have all the information of the partitions for
> > that corresponding topic , Please confirm
> >
> > Thanks
> >
> > On Fri, Feb 27, 2015 at 11:20 AM, Alex Melville <am...@g.hmc.edu>
> > wrote:
> >
> > > I was going to make a separate email thread for this question but this
> > > thread's topic echoes what my own would have been.
> > >
> > >
> > > How can I query a broker or zookeeper for the number of partitions in a
> > > given topic? I'm trying to write a custom partitioner that sends a
> > message
> > > to every partition within a topic, and so I need to know the total
> number
> > > of partitions before I call Producer.send().
> > >
> > >
> > > Alex
> > >
> > > On Thu, Feb 26, 2015 at 7:32 PM, tao xiao <xi...@gmail.com>
> wrote:
> > >
> > > > Gaurav,
> > > >
> > > > You can get the partition number the message belongs to via
> > > > MessageAndMetadata.partition()
> > > >
> > > > On Fri, Feb 27, 2015 at 5:16 AM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > The partition api is exposed to the consumer in 0.8.2.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Feb 26, 2015 at 10:53 AM, Gaurav Agarwal <
> > > gaurav130403@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > After retrieving a kafka stream or kafka message how to get the
> > > > > > corresponding partition number to which it belongs ? I am using
> > kafka
> > > > > > version 0.8.1.
> > > > > > More specifically kafka.consumer.KafkaStream and
> > > > > > kafka.message.MessageAndMetaData classes, does not provide API to
> > > > > retrieve
> > > > > > partition number. Are there any other API's to get the partition
> > > > number?
> > > > > >
> > > > > > IF there are multiple partitions of a topic ,Do i need to declare
> > > from
> > > > > java
> > > > > > code how many partitions the topic contains or i can leave it
> topic
> > > > > > Kafkastream will take the partition information from kafka broker
> > at
> > > > > > runtime.?
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Tao
> > > >
> > >
> >
>
--
Regards,
Tao
Re: kafka partitions api
Posted by Alex Melville <am...@g.hmc.edu>.
Tao and Gaurav,
After looking through the source code in Kafka v8.2.0, I don't see any
partition() function on the MessageAndMetadata object. Here's the class's
source:
package kafka.message
import kafka.serializer.Decoder
import kafka.utils.Utils
case class MessageAndMetadata[K, V](topic: String, partition: Int,
private val rawMessage: Message,
offset: Long,
keyDecoder: Decoder[K],
valueDecoder: Decoder[V]) {
/**
* Return the decoded message key and payload
*/
def key(): K = if(rawMessage.key == null) null.asInstanceOf[K] else
keyDecoder.fromBytes(Utils.readBytes(rawMessage.key))
def message(): V = if(rawMessage.isNull) null.asInstanceOf[V] else
valueDecoder.fromBytes(Utils.readBytes(rawMessage.payload))
-Alex M.
On Thu, Feb 26, 2015 at 9:54 PM, Gaurav Agarwal <ga...@gmail.com>
wrote:
> that's fine to me , you can open a separate thread , But the original
> question when the consumerconnector got connected to a separate topic ,
> Whether KafkaStream will have all the information of the partitions for
> that corresponding topic , Please confirm
>
> Thanks
>
> On Fri, Feb 27, 2015 at 11:20 AM, Alex Melville <am...@g.hmc.edu>
> wrote:
>
> > I was going to make a separate email thread for this question but this
> > thread's topic echoes what my own would have been.
> >
> >
> > How can I query a broker or zookeeper for the number of partitions in a
> > given topic? I'm trying to write a custom partitioner that sends a
> message
> > to every partition within a topic, and so I need to know the total number
> > of partitions before I call Producer.send().
> >
> >
> > Alex
> >
> > On Thu, Feb 26, 2015 at 7:32 PM, tao xiao <xi...@gmail.com> wrote:
> >
> > > Gaurav,
> > >
> > > You can get the partition number the message belongs to via
> > > MessageAndMetadata.partition()
> > >
> > > On Fri, Feb 27, 2015 at 5:16 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > The partition api is exposed to the consumer in 0.8.2.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Feb 26, 2015 at 10:53 AM, Gaurav Agarwal <
> > gaurav130403@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > After retrieving a kafka stream or kafka message how to get the
> > > > > corresponding partition number to which it belongs ? I am using
> kafka
> > > > > version 0.8.1.
> > > > > More specifically kafka.consumer.KafkaStream and
> > > > > kafka.message.MessageAndMetaData classes, does not provide API to
> > > > retrieve
> > > > > partition number. Are there any other API's to get the partition
> > > number?
> > > > >
> > > > > IF there are multiple partitions of a topic ,Do i need to declare
> > from
> > > > java
> > > > > code how many partitions the topic contains or i can leave it topic
> > > > > Kafkastream will take the partition information from kafka broker
> at
> > > > > runtime.?
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Tao
> > >
> >
>
Re: kafka partitions api
Posted by Gaurav Agarwal <ga...@gmail.com>.
that's fine to me , you can open a separate thread , But the original
question when the consumerconnector got connected to a separate topic ,
Whether KafkaStream will have all the information of the partitions for
that corresponding topic , Please confirm
Thanks
On Fri, Feb 27, 2015 at 11:20 AM, Alex Melville <am...@g.hmc.edu> wrote:
> I was going to make a separate email thread for this question but this
> thread's topic echoes what my own would have been.
>
>
> How can I query a broker or zookeeper for the number of partitions in a
> given topic? I'm trying to write a custom partitioner that sends a message
> to every partition within a topic, and so I need to know the total number
> of partitions before I call Producer.send().
>
>
> Alex
>
> On Thu, Feb 26, 2015 at 7:32 PM, tao xiao <xi...@gmail.com> wrote:
>
> > Gaurav,
> >
> > You can get the partition number the message belongs to via
> > MessageAndMetadata.partition()
> >
> > On Fri, Feb 27, 2015 at 5:16 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > The partition api is exposed to the consumer in 0.8.2.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Feb 26, 2015 at 10:53 AM, Gaurav Agarwal <
> gaurav130403@gmail.com
> > >
> > > wrote:
> > >
> > > > After retrieving a kafka stream or kafka message how to get the
> > > > corresponding partition number to which it belongs ? I am using kafka
> > > > version 0.8.1.
> > > > More specifically kafka.consumer.KafkaStream and
> > > > kafka.message.MessageAndMetaData classes, does not provide API to
> > > retrieve
> > > > partition number. Are there any other API's to get the partition
> > number?
> > > >
> > > > IF there are multiple partitions of a topic ,Do i need to declare
> from
> > > java
> > > > code how many partitions the topic contains or i can leave it topic
> > > > Kafkastream will take the partition information from kafka broker at
> > > > runtime.?
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Tao
> >
>
Re: kafka partitions api
Posted by Alex Melville <am...@g.hmc.edu>.
I was going to make a separate email thread for this question but this
thread's topic echoes what my own would have been.
How can I query a broker or zookeeper for the number of partitions in a
given topic? I'm trying to write a custom partitioner that sends a message
to every partition within a topic, and so I need to know the total number
of partitions before I call Producer.send().
Alex
On Thu, Feb 26, 2015 at 7:32 PM, tao xiao <xi...@gmail.com> wrote:
> Gaurav,
>
> You can get the partition number the message belongs to via
> MessageAndMetadata.partition()
>
> On Fri, Feb 27, 2015 at 5:16 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > The partition api is exposed to the consumer in 0.8.2.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Feb 26, 2015 at 10:53 AM, Gaurav Agarwal <gaurav130403@gmail.com
> >
> > wrote:
> >
> > > After retrieving a kafka stream or kafka message how to get the
> > > corresponding partition number to which it belongs ? I am using kafka
> > > version 0.8.1.
> > > More specifically kafka.consumer.KafkaStream and
> > > kafka.message.MessageAndMetaData classes, does not provide API to
> > retrieve
> > > partition number. Are there any other API's to get the partition
> number?
> > >
> > > IF there are multiple partitions of a topic ,Do i need to declare from
> > java
> > > code how many partitions the topic contains or i can leave it topic
> > > Kafkastream will take the partition information from kafka broker at
> > > runtime.?
> > >
> >
>
>
>
> --
> Regards,
> Tao
>
Re: kafka partitions api
Posted by tao xiao <xi...@gmail.com>.
Gaurav,
You can get the partition number the message belongs to via
MessageAndMetadata.partition()
On Fri, Feb 27, 2015 at 5:16 AM, Jun Rao <ju...@confluent.io> wrote:
> The partition api is exposed to the consumer in 0.8.2.
>
> Thanks,
>
> Jun
>
> On Thu, Feb 26, 2015 at 10:53 AM, Gaurav Agarwal <ga...@gmail.com>
> wrote:
>
> > After retrieving a kafka stream or kafka message how to get the
> > corresponding partition number to which it belongs ? I am using kafka
> > version 0.8.1.
> > More specifically kafka.consumer.KafkaStream and
> > kafka.message.MessageAndMetaData classes, does not provide API to
> retrieve
> > partition number. Are there any other API's to get the partition number?
> >
> > IF there are multiple partitions of a topic ,Do i need to declare from
> java
> > code how many partitions the topic contains or i can leave it topic
> > Kafkastream will take the partition information from kafka broker at
> > runtime.?
> >
>
--
Regards,
Tao
Re: kafka partitions api
Posted by Jun Rao <ju...@confluent.io>.
The partition api is exposed to the consumer in 0.8.2.
Thanks,
Jun
On Thu, Feb 26, 2015 at 10:53 AM, Gaurav Agarwal <ga...@gmail.com>
wrote:
> After retrieving a kafka stream or kafka message how to get the
> corresponding partition number to which it belongs ? I am using kafka
> version 0.8.1.
> More specifically kafka.consumer.KafkaStream and
> kafka.message.MessageAndMetaData classes, does not provide API to retrieve
> partition number. Are there any other API's to get the partition number?
>
> IF there are multiple partitions of a topic ,Do i need to declare from java
> code how many partitions the topic contains or i can leave it topic
> Kafkastream will take the partition information from kafka broker at
> runtime.?
>
Fwd: kafka partitions api
Posted by Gaurav Agarwal <ga...@gmail.com>.
Hello
After retrieving a kafka stream or kafka message how to get the
corresponding partition number to which it belongs ? I am using kafka
version 0.8.1.
More specifically kafka.consumer.KafkaStream and
kafka.message.MessageAndMetaData classes, does not provide API to retrieve
partition number. Are there any other API's to get the partition number?
IF there are multiple partitions of a topic ,Do i need to declare from java
code how many partitions the topic contains or i can leave it topic
Kafkastream will take the partition information from kafka broker at
runtime.?