You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by joe smith <wa...@yahoo.com> on 2014/04/03 18:06:46 UTC
DefaultEventHandler's getPartition's return value
Hi,
My apologies - a newbie here.
In looking at the getPartition() method, I notice the returned value for keyed message is the "index" of a partition. But when there is no key, then the value returned is the "id" of a partition.
case None =>
val availablePartitions = topicPartitionList.filter(_.leaderBrokerIdOpt.isDefined)
if (availablePartitions.isEmpty)
throw new LeaderNotAvailableException("No leader for any partition in topic " + topic)
val index = Utils.abs(Random.nextInt) % availablePartitions.size
val partitionId = availablePartitions(index).partitionId
sendPartitionPerTopicCache.put(topic, partitionId)
<< partitionId >>
It probably won't make a difference when there is no gaps in the sequence of partitions. However, when there is a gap - e.g. some partitions are not available - then the partition id and index would be different.
In the partitionAndCollate() method, the semantic seems to be using index instead of id.
Thanks
RE: DefaultEventHandler's getPartition's return value
Posted by "Wang, Jason" <ja...@teamaol.com>.
Thanks Jun and Neha.
-j
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com]
Sent: Friday, April 04, 2014 12:07 PM
To: users@kafka.apache.org
Subject: Re: DefaultEventHandler's getPartition's return value
The new producer is org.apache.kafka.clients.producer.KafkaProducer in
trunk.
Thanks,
Neha
On Apr 4, 2014 8:31 AM, "joe smith" <wa...@yahoo.com> wrote:
> Jun,
>
> Thank you. I'm looking at the 0.8.1 -
> kafka.producer.async.DefaultEventHandler. Would you please point to the
> new producer that you are referring to. I assume that's one we should be
> using.
>
> Thanks, again,
> -j
>
>
> On Thursday, April 3, 2014 11:20 PM, Jun Rao <ju...@gmail.com> wrote:
>
> Yes, it's not very intuitive. Currently, it is true that we don't have gap
> in the partition ids. Since we already build the new producer, we probably
> don't have to patch the old producer.
>
> Thanks,
>
> Jun
>
>
>
> On Thu, Apr 3, 2014 at 9:06 AM, joe smith <wa...@yahoo.com> wrote:
>
> > Hi,
> >
> > My apologies - a newbie here.
> >
> > In looking at the getPartition() method, I notice the returned value for
> > keyed message is the "index" of a partition. But when there is no key,
> > then the value returned is the "id" of a partition.
> >
> > case None =>
> > val availablePartitions =
> > topicPartitionList.filter(_.leaderBrokerIdOpt.isDefined)
> > if (availablePartitions.isEmpty)
> > throw new LeaderNotAvailableException("No leader for any
> > partition in topic " + topic)
> > val index = Utils.abs(Random.nextInt) %
> > availablePartitions.size
> > val partitionId = availablePartitions(index).partitionId
> > sendPartitionPerTopicCache.put(topic, partitionId)
> > << partitionId >>
> >
> > It probably won't make a difference when there is no gaps in the sequence
> > of partitions. However, when there is a gap - e.g. some partitions are
> not
> > available - then the partition id and index would be different.
> >
> > In the partitionAndCollate() method, the semantic seems to be using index
> > instead of id.
> >
> > Thanks
Re: DefaultEventHandler's getPartition's return value
Posted by Neha Narkhede <ne...@gmail.com>.
The new producer is org.apache.kafka.clients.producer.KafkaProducer in
trunk.
Thanks,
Neha
On Apr 4, 2014 8:31 AM, "joe smith" <wa...@yahoo.com> wrote:
> Jun,
>
> Thank you. I'm looking at the 0.8.1 -
> kafka.producer.async.DefaultEventHandler. Would you please point to the
> new producer that you are referring to. I assume that's one we should be
> using.
>
> Thanks, again,
> -j
>
>
> On Thursday, April 3, 2014 11:20 PM, Jun Rao <ju...@gmail.com> wrote:
>
> Yes, it's not very intuitive. Currently, it is true that we don't have gap
> in the partition ids. Since we already build the new producer, we probably
> don't have to patch the old producer.
>
> Thanks,
>
> Jun
>
>
>
> On Thu, Apr 3, 2014 at 9:06 AM, joe smith <wa...@yahoo.com> wrote:
>
> > Hi,
> >
> > My apologies - a newbie here.
> >
> > In looking at the getPartition() method, I notice the returned value for
> > keyed message is the "index" of a partition. But when there is no key,
> > then the value returned is the "id" of a partition.
> >
> > case None =>
> > val availablePartitions =
> > topicPartitionList.filter(_.leaderBrokerIdOpt.isDefined)
> > if (availablePartitions.isEmpty)
> > throw new LeaderNotAvailableException("No leader for any
> > partition in topic " + topic)
> > val index = Utils.abs(Random.nextInt) %
> > availablePartitions.size
> > val partitionId = availablePartitions(index).partitionId
> > sendPartitionPerTopicCache.put(topic, partitionId)
> > << partitionId >>
> >
> > It probably won't make a difference when there is no gaps in the sequence
> > of partitions. However, when there is a gap - e.g. some partitions are
> not
> > available - then the partition id and index would be different.
> >
> > In the partitionAndCollate() method, the semantic seems to be using index
> > instead of id.
> >
> > Thanks
Re: DefaultEventHandler's getPartition's return value
Posted by Jun Rao <ju...@gmail.com>.
The new producer is only in trunk, under clients/. It's written in java.
Thanks,
Jun
On Fri, Apr 4, 2014 at 8:31 AM, joe smith <wa...@yahoo.com> wrote:
> Jun,
>
> Thank you. I'm looking at the 0.8.1 -
> kafka.producer.async.DefaultEventHandler. Would you please point to the
> new producer that you are referring to. I assume that's one we should be
> using.
>
> Thanks, again,
> -j
>
>
> On Thursday, April 3, 2014 11:20 PM, Jun Rao <ju...@gmail.com> wrote:
>
> Yes, it's not very intuitive. Currently, it is true that we don't have gap
> in the partition ids. Since we already build the new producer, we probably
> don't have to patch the old producer.
>
> Thanks,
>
> Jun
>
>
>
> On Thu, Apr 3, 2014 at 9:06 AM, joe smith <wa...@yahoo.com> wrote:
>
> > Hi,
> >
> > My apologies - a newbie here.
> >
> > In looking at the getPartition() method, I notice the returned value for
> > keyed message is the "index" of a partition. But when there is no key,
> > then the value returned is the "id" of a partition.
> >
> > case None =>
> > val availablePartitions =
> > topicPartitionList.filter(_.leaderBrokerIdOpt.isDefined)
> > if (availablePartitions.isEmpty)
> > throw new LeaderNotAvailableException("No leader for any
> > partition in topic " + topic)
> > val index = Utils.abs(Random.nextInt) %
> > availablePartitions.size
> > val partitionId = availablePartitions(index).partitionId
> > sendPartitionPerTopicCache.put(topic, partitionId)
> > << partitionId >>
> >
> > It probably won't make a difference when there is no gaps in the sequence
> > of partitions. However, when there is a gap - e.g. some partitions are
> not
> > available - then the partition id and index would be different.
> >
> > In the partitionAndCollate() method, the semantic seems to be using index
> > instead of id.
> >
> > Thanks
>
Re: DefaultEventHandler's getPartition's return value
Posted by joe smith <wa...@yahoo.com>.
Jun,
Thank you. I'm looking at the 0.8.1 - kafka.producer.async.DefaultEventHandler. Would you please point to the new producer that you are referring to. I assume that's one we should be using.
Thanks, again,
-j
On Thursday, April 3, 2014 11:20 PM, Jun Rao <ju...@gmail.com> wrote:
Yes, it's not very intuitive. Currently, it is true that we don't have gap
in the partition ids. Since we already build the new producer, we probably
don't have to patch the old producer.
Thanks,
Jun
On Thu, Apr 3, 2014 at 9:06 AM, joe smith <wa...@yahoo.com> wrote:
> Hi,
>
> My apologies - a newbie here.
>
> In looking at the getPartition() method, I notice the returned value for
> keyed message is the "index" of a partition. But when there is no key,
> then the value returned is the "id" of a partition.
>
> case None =>
> val availablePartitions =
> topicPartitionList.filter(_.leaderBrokerIdOpt.isDefined)
> if (availablePartitions.isEmpty)
> throw new LeaderNotAvailableException("No leader for any
> partition in topic " + topic)
> val index = Utils.abs(Random.nextInt) %
> availablePartitions.size
> val partitionId = availablePartitions(index).partitionId
> sendPartitionPerTopicCache.put(topic, partitionId)
> << partitionId >>
>
> It probably won't make a difference when there is no gaps in the sequence
> of partitions. However, when there is a gap - e.g. some partitions are not
> available - then the partition id and index would be different.
>
> In the partitionAndCollate() method, the semantic seems to be using index
> instead of id.
>
> Thanks
Re: DefaultEventHandler's getPartition's return value
Posted by Jun Rao <ju...@gmail.com>.
Yes, it's not very intuitive. Currently, it is true that we don't have gap
in the partition ids. Since we already build the new producer, we probably
don't have to patch the old producer.
Thanks,
Jun
On Thu, Apr 3, 2014 at 9:06 AM, joe smith <wa...@yahoo.com> wrote:
> Hi,
>
> My apologies - a newbie here.
>
> In looking at the getPartition() method, I notice the returned value for
> keyed message is the "index" of a partition. But when there is no key,
> then the value returned is the "id" of a partition.
>
> case None =>
> val availablePartitions =
> topicPartitionList.filter(_.leaderBrokerIdOpt.isDefined)
> if (availablePartitions.isEmpty)
> throw new LeaderNotAvailableException("No leader for any
> partition in topic " + topic)
> val index = Utils.abs(Random.nextInt) %
> availablePartitions.size
> val partitionId = availablePartitions(index).partitionId
> sendPartitionPerTopicCache.put(topic, partitionId)
> << partitionId >>
>
> It probably won't make a difference when there is no gaps in the sequence
> of partitions. However, when there is a gap - e.g. some partitions are not
> available - then the partition id and index would be different.
>
> In the partitionAndCollate() method, the semantic seems to be using index
> instead of id.
>
> Thanks