Questions on the new Kafka consumer

[Rajiv Kurian <ra...@signalfx.com>]

I was reading the documentation for the new Kafka consumer API at
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
and came across this:

"Each Kafka consumer must specify a consumer group that it belongs to."

Currently we use Kafka but WITHOUT the consumer group functionality since I
do the partition distribution based on my own app logic. I also persist my
own offsets in an external store.
I also sometimes need to consume from the beginning of a partition or just
some arbitrary offset. Currently I use the SimpleConsumer to do this. I
just give it the topics and partitions I want to consume from and I
maintain the partition -> broker mapping myself. I also maintain the offset
myself and when I need to consume from the beginning of a partition I ask
the broker for the earliest offset and start consuming from there.

I am glad that the new consumer does the broker -> partition mapping itself
and I can get rid of a lot of code if I transitioned to it. But given that
the documentation suggests that I have to use the Consumer group, it
doesn't seem that it will support the lower level features that I need. I
do see the commitSync method which suggests that I am in control of the
offset management and for my use case I could just never call it. Is there
also a way for me to specify which partition to consume from exactly (in
addition to the topic) and also from which offset?

Thanks,
Rajiv

RE: Questions on the new Kafka consumer

[pr...@wipro.com]

Hi,

According to the wiki: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design, allowing manual partition and topic access is a design goal.

Also the new API has functions to seek and subscribe at the partition level: http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/

This for the KafkaConsumer from kafka version 0.9.

Prajod

-----Original Message-----
From: Rajiv Kurian [mailto:rajiv@signalfx.com]
Sent: 14 October 2015 01:44
To: users@kafka.apache.org
Subject: Questions on the new Kafka consumer

I was reading the documentation for the new Kafka consumer API at https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
and came across this:

"Each Kafka consumer must specify a consumer group that it belongs to."

Currently we use Kafka but WITHOUT the consumer group functionality since I do the partition distribution based on my own app logic. I also persist my own offsets in an external store.
I also sometimes need to consume from the beginning of a partition or just some arbitrary offset. Currently I use the SimpleConsumer to do this. I just give it the topics and partitions I want to consume from and I maintain the partition -> broker mapping myself. I also maintain the offset myself and when I need to consume from the beginning of a partition I ask the broker for the earliest offset and start consuming from there.

I am glad that the new consumer does the broker -> partition mapping itself and I can get rid of a lot of code if I transitioned to it. But given that the documentation suggests that I have to use the Consumer group, it doesn't seem that it will support the lower level features that I need. I do see the commitSync method which suggests that I am in control of the offset management and for my use case I could just never call it. Is there also a way for me to specify which partition to consume from exactly (in addition to the topic) and also from which offset?

Thanks,
Rajiv
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com

Re: Questions on the new Kafka consumer

[Rajiv Kurian <ra...@signalfx.com>]

Aah I didn't read long enough. I just read the subscribing to specific
partitions, managing your own offsets and the controlling the consumer's
position sections.

So a follow up question - when I am subscribing to specific partitions and
managing my own offsets and controlling the consumer's position what does
the consumer group do? Do I still have to specify it all the time?

Thanks,
Rajiv

On Tue, Oct 13, 2015 at 1:14 PM, Rajiv Kurian <ra...@signalfx.com> wrote:

> I was reading the documentation for the new Kafka consumer API at
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java
> and came across this:
>
> "Each Kafka consumer must specify a consumer group that it belongs to."
>
> Currently we use Kafka but WITHOUT the consumer group functionality since
> I do the partition distribution based on my own app logic. I also persist
> my own offsets in an external store.
> I also sometimes need to consume from the beginning of a partition or just
> some arbitrary offset. Currently I use the SimpleConsumer to do this. I
> just give it the topics and partitions I want to consume from and I
> maintain the partition -> broker mapping myself. I also maintain the offset
> myself and when I need to consume from the beginning of a partition I ask
> the broker for the earliest offset and start consuming from there.
>
> I am glad that the new consumer does the broker -> partition mapping
> itself and I can get rid of a lot of code if I transitioned to it. But
> given that the documentation suggests that I have to use the Consumer
> group, it doesn't seem that it will support the lower level features that I
> need. I do see the commitSync method which suggests that I am in control of
> the offset management and for my use case I could just never call it. Is
> there also a way for me to specify which partition to consume from exactly
> (in addition to the topic) and also from which offset?
>
> Thanks,
> Rajiv
>