You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jason Gustafson <ja...@confluent.io> on 2016/01/11 18:52:05 UTC
Re: Kafka 0.9 Consumer Group
Hi Howard,
The offsets are persisted in the __consumer_offsets topic indefinitely.
Since you're using manual commit, have you ensured that auto.offset.reset
is disabled? It might also help if you provide a little more detail on how
you're verifying that offsets were lost.
-Jason
On Mon, Jan 11, 2016 at 7:42 AM, Wang, Howard <Ho...@washpost.com>
wrote:
> Hi,
>
> I have a question regarding the Kafka 0.9 Consumer Group . I manually
> commit offsets using the Kafka 0.9 Consumer created with a consumer group.
>
> However, after my app restarted totally from scratch, the consumer group
> seems to lose all the offsets. Is that true that the consumer offsets are
> transient and will be gone after the consumer group has no member and gets
> deleted?
>
> Thanks.
>
> Howard
> --
> Howard Wang
> Engineering - Big Data and Personalization
> Washington Post Media
>
> 1150 15th St NW, Washington, DC 20071
> p. 202-334-9195
> Email: howard.wang@washpost.com
>
Re: Kafka 0.9 Consumer Group
Posted by Jason Gustafson <ja...@confluent.io>.
Ah, that actually makes sense. The consumer-groups.sh script only returns
offset data when the group is active. The offsets should still be there,
but it does seem unfortunate that there's no way to view them. We have
KAFKA-3059 which adds some additional capabilities for managing offsets
with this script, so maybe we can extend it to provide this feature as well.
-Jason
On Mon, Jan 11, 2016 at 11:33 AM, Wang, Howard <Ho...@washpost.com>
wrote:
> Hi Jason,
>
> I used the kafka-consumer-groups.sh to check my consumer group :
> ~/GitHub/kafka/bin/kafka-consumer-groups.sh --bootstrap-server <my server
> name> ‹group test.group --describe --new-consumer .
>
> I ran this command several times after my app was shut down. I always get
> "Consumer group `test.group` does not exist or is rebalancing.² response.
>
>
>
> I did set the enable.auto.commit to false. Below is how I set my
> KafkaConsumer.
>
> Properties props = new Properties();
> props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
> myAppConfig.getKafkaBroker());
> props.put(ConsumerConfig.GROUP_ID_CONFIG,
> myAppConfig.getKafkaConsumerGroup());
> props.put(ConsumerConfig.CLIENT_ID_CONFIG,
> myAppConfig.getRandomNodeId());
> props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
> props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");
> props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG,
> myAppConfig.getKafkaConsumerSessionTimeout());
> props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
> "org.apache.kafka.common.serialization.StringDeserializer");
> props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
> "org.apache.kafka.common.serialization.StringDeserializer");
> consumer = new KafkaConsumer<>(props);
>
>
>
>
> Thanks.
>
> Howard
>
> On 1/11/16, 12:55 PM, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> >Sorry, wrong property, I meant enable.auto.commit.
> >
> >-Jason
> >
> >On Mon, Jan 11, 2016 at 9:52 AM, Jason Gustafson <ja...@confluent.io>
> >wrote:
> >
> >> Hi Howard,
> >>
> >> The offsets are persisted in the __consumer_offsets topic indefinitely.
> >> Since you're using manual commit, have you ensured that
> >>auto.offset.reset
> >> is disabled? It might also help if you provide a little more detail on
> >>how
> >> you're verifying that offsets were lost.
> >>
> >> -Jason
> >>
> >> On Mon, Jan 11, 2016 at 7:42 AM, Wang, Howard <Howard.Wang@washpost.com
> >
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I have a question regarding the Kafka 0.9 Consumer Group . I manually
> >>> commit offsets using the Kafka 0.9 Consumer created with a consumer
> >>>group.
> >>>
> >>> However, after my app restarted totally from scratch, the consumer
> >>>group
> >>> seems to lose all the offsets. Is that true that the consumer offsets
> >>>are
> >>> transient and will be gone after the consumer group has no member and
> >>>gets
> >>> deleted?
> >>>
> >>> Thanks.
> >>>
> >>> Howard
> >>> --
> >>> Howard Wang
> >>> Engineering - Big Data and Personalization
> >>> Washington Post Media
> >>>
> >>> 1150 15th St NW, Washington, DC 20071
> >>> p. 202-334-9195
> >>> Email: howard.wang@washpost.com
> >>>
> >>
> >>
>
>
Re: Kafka 0.9 Consumer Group
Posted by "Wang, Howard" <Ho...@washpost.com>.
Hi Jason,
I used the kafka-consumer-groups.sh to check my consumer group :
~/GitHub/kafka/bin/kafka-consumer-groups.sh --bootstrap-server <my server
name> ‹group test.group --describe --new-consumer .
I ran this command several times after my app was shut down. I always get
"Consumer group `test.group` does not exist or is rebalancing.² response.
I did set the enable.auto.commit to false. Below is how I set my
KafkaConsumer.
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
myAppConfig.getKafkaBroker());
props.put(ConsumerConfig.GROUP_ID_CONFIG,
myAppConfig.getKafkaConsumerGroup());
props.put(ConsumerConfig.CLIENT_ID_CONFIG,
myAppConfig.getRandomNodeId());
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG,
myAppConfig.getKafkaConsumerSessionTimeout());
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
"org.apache.kafka.common.serialization.StringDeserializer");
consumer = new KafkaConsumer<>(props);
Thanks.
Howard
On 1/11/16, 12:55 PM, "Jason Gustafson" <ja...@confluent.io> wrote:
>Sorry, wrong property, I meant enable.auto.commit.
>
>-Jason
>
>On Mon, Jan 11, 2016 at 9:52 AM, Jason Gustafson <ja...@confluent.io>
>wrote:
>
>> Hi Howard,
>>
>> The offsets are persisted in the __consumer_offsets topic indefinitely.
>> Since you're using manual commit, have you ensured that
>>auto.offset.reset
>> is disabled? It might also help if you provide a little more detail on
>>how
>> you're verifying that offsets were lost.
>>
>> -Jason
>>
>> On Mon, Jan 11, 2016 at 7:42 AM, Wang, Howard <Ho...@washpost.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a question regarding the Kafka 0.9 Consumer Group . I manually
>>> commit offsets using the Kafka 0.9 Consumer created with a consumer
>>>group.
>>>
>>> However, after my app restarted totally from scratch, the consumer
>>>group
>>> seems to lose all the offsets. Is that true that the consumer offsets
>>>are
>>> transient and will be gone after the consumer group has no member and
>>>gets
>>> deleted?
>>>
>>> Thanks.
>>>
>>> Howard
>>> --
>>> Howard Wang
>>> Engineering - Big Data and Personalization
>>> Washington Post Media
>>>
>>> 1150 15th St NW, Washington, DC 20071
>>> p. 202-334-9195
>>> Email: howard.wang@washpost.com
>>>
>>
>>
Re: Kafka 0.9 Consumer Group
Posted by Jason Gustafson <ja...@confluent.io>.
Sorry, wrong property, I meant enable.auto.commit.
-Jason
On Mon, Jan 11, 2016 at 9:52 AM, Jason Gustafson <ja...@confluent.io> wrote:
> Hi Howard,
>
> The offsets are persisted in the __consumer_offsets topic indefinitely.
> Since you're using manual commit, have you ensured that auto.offset.reset
> is disabled? It might also help if you provide a little more detail on how
> you're verifying that offsets were lost.
>
> -Jason
>
> On Mon, Jan 11, 2016 at 7:42 AM, Wang, Howard <Ho...@washpost.com>
> wrote:
>
>> Hi,
>>
>> I have a question regarding the Kafka 0.9 Consumer Group . I manually
>> commit offsets using the Kafka 0.9 Consumer created with a consumer group.
>>
>> However, after my app restarted totally from scratch, the consumer group
>> seems to lose all the offsets. Is that true that the consumer offsets are
>> transient and will be gone after the consumer group has no member and gets
>> deleted?
>>
>> Thanks.
>>
>> Howard
>> --
>> Howard Wang
>> Engineering - Big Data and Personalization
>> Washington Post Media
>>
>> 1150 15th St NW, Washington, DC 20071
>> p. 202-334-9195
>> Email: howard.wang@washpost.com
>>
>
>