You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Josh J <jo...@gmail.com> on 2014/08/14 10:46:29 UTC
consumer read from specific partition
Hi,
Suppose I have N partitions. I would like to have X different consumer
threads ( X < N) read from a specified set of partitions. How can I achieve
this?
Thanks,
Josh
Re: consumer read from specific partition
Posted by Jonathan Weeks <jo...@gmail.com>.
The high level API gives you access to the raw Kafka event which is MessageAndMetadata, which has two pieces: key and actual message.
Best Regards,
-Jonathan
On Aug 18, 2014, at 9:50 AM, Josh J <jo...@gmail.com> wrote:
>> One tactic that might be worth exploring is to rely on the message key to
> facilitate this.
>
>> It would require engineering careful functions for the key which hashes
> to the partitions for your topic(s). It would also mean that your consumers
> for the topic would be evaluating the key and discarding messages that
> aren’t relevant.
>
>> The only other option I can think of if you are using the high-level API
> would be finer-grained topics.
>
> Yes, this works great ! My producer is bucketing the messages based on the
> key (the key is a timestamp and I simply mod the timestamp as a function of
> the number of buckets). I can then do the same function on the consumer
> when it reads the key. I'm essentially implementing consumer sliding
> window. Any suggestions or tips on where I would implement reading the
> message key?
>
> Thanks,
> Josh
>
>
> On Mon, Aug 18, 2014 at 6:43 PM, Jonathan Weeks <jo...@gmail.com>
> wrote:
>
>> One tactic that might be worth exploring is to rely on the message key to
>> facilitate this.
>>
>> It would require engineering careful functions for the key which hashes to
>> the partitions for your topic(s). It would also mean that your consumers
>> for the topic would be evaluating the key and discarding messages that
>> aren’t relevant.
>>
>> The only other option I can think of if you are using the high-level API
>> would be finer-grained topics.
>>
>> Best Regards,
>>
>> -Jonathan
>>
>> On Aug 18, 2014, at 9:14 AM, Josh J <jo...@gmail.com> wrote:
>>
>>> Is it possible to modify and use the high level consumer so that I can
>>> ignore processing certain partitions?
>>>
>>>
>>> On Mon, Aug 18, 2014 at 5:07 PM, Sharninder <sh...@gmail.com>
>> wrote:
>>>
>>>> On Mon, Aug 18, 2014 at 7:27 PM, Josh J <jo...@gmail.com> wrote:
>>>>
>>>>>> You can see an example of using the SimpleConsumer here
>>>>> <
>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>>>>>>
>>>>>
>>>>> Any suggestions on where in the code to modify the high level producer
>> to
>>>>> support reading from specific partitions ?
>>>>>
>>>>>
>>>> High level producer? I'm assuming you meant to write the high level
>>>> consumer, in which case it isn't possible. The link above, which has an
>>>> example for reading messages off a specific partition, is for the Simple
>>>> consumer, which ironically, is more complex than the high level
>> consumer.
>>>>
>>>> In short, if you have a usecase where you want to read from a specific
>>>> partition, you will need to implement a simple consumer.
>>>>
>>>> --
>>>> Sharninder
>>>>
>>>> Josh
>>>>> .
>>>>>
>>>>> On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede <
>> neha.narkhede@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> You can see an example of using the SimpleConsumer here
>>>>>> <
>>>>>>
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>>>>>>>
>>>>>> .
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sh...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>> Implement the low level "Simple Consumer".
>>>>>>>
>>>>>>> --
>>>>>>> Sharninder
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 14, 2014 at 2:16 PM, Josh J <jo...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Suppose I have N partitions. I would like to have X different
>>>>> consumer
>>>>>>>> threads ( X < N) read from a specified set of partitions. How can I
>>>>>>> achieve
>>>>>>>> this?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Josh
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
Re: consumer read from specific partition
Posted by Josh J <jo...@gmail.com>.
> One tactic that might be worth exploring is to rely on the message key to
facilitate this.
> It would require engineering careful functions for the key which hashes
to the partitions for your topic(s). It would also mean that your consumers
for the topic would be evaluating the key and discarding messages that
aren’t relevant.
> The only other option I can think of if you are using the high-level API
would be finer-grained topics.
Yes, this works great ! My producer is bucketing the messages based on the
key (the key is a timestamp and I simply mod the timestamp as a function of
the number of buckets). I can then do the same function on the consumer
when it reads the key. I'm essentially implementing consumer sliding
window. Any suggestions or tips on where I would implement reading the
message key?
Thanks,
Josh
On Mon, Aug 18, 2014 at 6:43 PM, Jonathan Weeks <jo...@gmail.com>
wrote:
> One tactic that might be worth exploring is to rely on the message key to
> facilitate this.
>
> It would require engineering careful functions for the key which hashes to
> the partitions for your topic(s). It would also mean that your consumers
> for the topic would be evaluating the key and discarding messages that
> aren’t relevant.
>
> The only other option I can think of if you are using the high-level API
> would be finer-grained topics.
>
> Best Regards,
>
> -Jonathan
>
> On Aug 18, 2014, at 9:14 AM, Josh J <jo...@gmail.com> wrote:
>
> > Is it possible to modify and use the high level consumer so that I can
> > ignore processing certain partitions?
> >
> >
> > On Mon, Aug 18, 2014 at 5:07 PM, Sharninder <sh...@gmail.com>
> wrote:
> >
> >> On Mon, Aug 18, 2014 at 7:27 PM, Josh J <jo...@gmail.com> wrote:
> >>
> >>>> You can see an example of using the SimpleConsumer here
> >>> <
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> >>>>
> >>>
> >>> Any suggestions on where in the code to modify the high level producer
> to
> >>> support reading from specific partitions ?
> >>>
> >>>
> >> High level producer? I'm assuming you meant to write the high level
> >> consumer, in which case it isn't possible. The link above, which has an
> >> example for reading messages off a specific partition, is for the Simple
> >> consumer, which ironically, is more complex than the high level
> consumer.
> >>
> >> In short, if you have a usecase where you want to read from a specific
> >> partition, you will need to implement a simple consumer.
> >>
> >> --
> >> Sharninder
> >>
> >> Josh
> >>> .
> >>>
> >>> On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede <
> neha.narkhede@gmail.com>
> >>> wrote:
> >>>
> >>>> You can see an example of using the SimpleConsumer here
> >>>> <
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> >>>>>
> >>>> .
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sh...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Implement the low level "Simple Consumer".
> >>>>>
> >>>>> --
> >>>>> Sharninder
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Aug 14, 2014 at 2:16 PM, Josh J <jo...@gmail.com> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> Suppose I have N partitions. I would like to have X different
> >>> consumer
> >>>>>> threads ( X < N) read from a specified set of partitions. How can I
> >>>>> achieve
> >>>>>> this?
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Josh
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: consumer read from specific partition
Posted by Jonathan Weeks <jo...@gmail.com>.
One tactic that might be worth exploring is to rely on the message key to facilitate this.
It would require engineering careful functions for the key which hashes to the partitions for your topic(s). It would also mean that your consumers for the topic would be evaluating the key and discarding messages that aren’t relevant.
The only other option I can think of if you are using the high-level API would be finer-grained topics.
Best Regards,
-Jonathan
On Aug 18, 2014, at 9:14 AM, Josh J <jo...@gmail.com> wrote:
> Is it possible to modify and use the high level consumer so that I can
> ignore processing certain partitions?
>
>
> On Mon, Aug 18, 2014 at 5:07 PM, Sharninder <sh...@gmail.com> wrote:
>
>> On Mon, Aug 18, 2014 at 7:27 PM, Josh J <jo...@gmail.com> wrote:
>>
>>>> You can see an example of using the SimpleConsumer here
>>> <
>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>>>>
>>>
>>> Any suggestions on where in the code to modify the high level producer to
>>> support reading from specific partitions ?
>>>
>>>
>> High level producer? I'm assuming you meant to write the high level
>> consumer, in which case it isn't possible. The link above, which has an
>> example for reading messages off a specific partition, is for the Simple
>> consumer, which ironically, is more complex than the high level consumer.
>>
>> In short, if you have a usecase where you want to read from a specific
>> partition, you will need to implement a simple consumer.
>>
>> --
>> Sharninder
>>
>> Josh
>>> .
>>>
>>> On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede <ne...@gmail.com>
>>> wrote:
>>>
>>>> You can see an example of using the SimpleConsumer here
>>>> <
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>>>>>
>>>> .
>>>>
>>>>
>>>>
>>>> On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sh...@gmail.com>
>>> wrote:
>>>>
>>>>> Implement the low level "Simple Consumer".
>>>>>
>>>>> --
>>>>> Sharninder
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 14, 2014 at 2:16 PM, Josh J <jo...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Suppose I have N partitions. I would like to have X different
>>> consumer
>>>>>> threads ( X < N) read from a specified set of partitions. How can I
>>>>> achieve
>>>>>> this?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Josh
>>>>>>
>>>>>
>>>>
>>>
>>
Re: consumer read from specific partition
Posted by Josh J <jo...@gmail.com>.
Is it possible to modify and use the high level consumer so that I can
ignore processing certain partitions?
On Mon, Aug 18, 2014 at 5:07 PM, Sharninder <sh...@gmail.com> wrote:
> On Mon, Aug 18, 2014 at 7:27 PM, Josh J <jo...@gmail.com> wrote:
>
> > > You can see an example of using the SimpleConsumer here
> > <
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > >
> >
> > Any suggestions on where in the code to modify the high level producer to
> > support reading from specific partitions ?
> >
> >
> High level producer? I'm assuming you meant to write the high level
> consumer, in which case it isn't possible. The link above, which has an
> example for reading messages off a specific partition, is for the Simple
> consumer, which ironically, is more complex than the high level consumer.
>
> In short, if you have a usecase where you want to read from a specific
> partition, you will need to implement a simple consumer.
>
> --
> Sharninder
>
> Josh
> > .
> >
> > On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede <ne...@gmail.com>
> > wrote:
> >
> > > You can see an example of using the SimpleConsumer here
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > > >
> > > .
> > >
> > >
> > >
> > > On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sh...@gmail.com>
> > wrote:
> > >
> > > > Implement the low level "Simple Consumer".
> > > >
> > > > --
> > > > Sharninder
> > > >
> > > >
> > > >
> > > > On Thu, Aug 14, 2014 at 2:16 PM, Josh J <jo...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Suppose I have N partitions. I would like to have X different
> > consumer
> > > > > threads ( X < N) read from a specified set of partitions. How can I
> > > > achieve
> > > > > this?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Josh
> > > > >
> > > >
> > >
> >
>
Re: consumer read from specific partition
Posted by Sharninder <sh...@gmail.com>.
On Mon, Aug 18, 2014 at 7:27 PM, Josh J <jo...@gmail.com> wrote:
> > You can see an example of using the SimpleConsumer here
> <
>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> >
>
> Any suggestions on where in the code to modify the high level producer to
> support reading from specific partitions ?
>
>
High level producer? I'm assuming you meant to write the high level
consumer, in which case it isn't possible. The link above, which has an
example for reading messages off a specific partition, is for the Simple
consumer, which ironically, is more complex than the high level consumer.
In short, if you have a usecase where you want to read from a specific
partition, you will need to implement a simple consumer.
--
Sharninder
Josh
> .
>
> On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede <ne...@gmail.com>
> wrote:
>
> > You can see an example of using the SimpleConsumer here
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> > >
> > .
> >
> >
> >
> > On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sh...@gmail.com>
> wrote:
> >
> > > Implement the low level "Simple Consumer".
> > >
> > > --
> > > Sharninder
> > >
> > >
> > >
> > > On Thu, Aug 14, 2014 at 2:16 PM, Josh J <jo...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Suppose I have N partitions. I would like to have X different
> consumer
> > > > threads ( X < N) read from a specified set of partitions. How can I
> > > achieve
> > > > this?
> > > >
> > > > Thanks,
> > > >
> > > > Josh
> > > >
> > >
> >
>
Re: consumer read from specific partition
Posted by Josh J <jo...@gmail.com>.
> You can see an example of using the SimpleConsumer here
<
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>
Any suggestions on where in the code to modify the high level producer to
support reading from specific partitions ?
Thanks,
Josh
.
On Thu, Aug 14, 2014 at 4:27 PM, Neha Narkhede <ne...@gmail.com>
wrote:
> You can see an example of using the SimpleConsumer here
> <
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> >
> .
>
>
>
> On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sh...@gmail.com> wrote:
>
> > Implement the low level "Simple Consumer".
> >
> > --
> > Sharninder
> >
> >
> >
> > On Thu, Aug 14, 2014 at 2:16 PM, Josh J <jo...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Suppose I have N partitions. I would like to have X different consumer
> > > threads ( X < N) read from a specified set of partitions. How can I
> > achieve
> > > this?
> > >
> > > Thanks,
> > >
> > > Josh
> > >
> >
>
Re: consumer read from specific partition
Posted by Neha Narkhede <ne...@gmail.com>.
You can see an example of using the SimpleConsumer here
<https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example>
.
On Thu, Aug 14, 2014 at 3:23 AM, Sharninder <sh...@gmail.com> wrote:
> Implement the low level "Simple Consumer".
>
> --
> Sharninder
>
>
>
> On Thu, Aug 14, 2014 at 2:16 PM, Josh J <jo...@gmail.com> wrote:
>
> > Hi,
> >
> > Suppose I have N partitions. I would like to have X different consumer
> > threads ( X < N) read from a specified set of partitions. How can I
> achieve
> > this?
> >
> > Thanks,
> >
> > Josh
> >
>
Re: consumer read from specific partition
Posted by Sharninder <sh...@gmail.com>.
Implement the low level "Simple Consumer".
--
Sharninder
On Thu, Aug 14, 2014 at 2:16 PM, Josh J <jo...@gmail.com> wrote:
> Hi,
>
> Suppose I have N partitions. I would like to have X different consumer
> threads ( X < N) read from a specified set of partitions. How can I achieve
> this?
>
> Thanks,
>
> Josh
>