You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jun MA <mj...@gmail.com> on 2016/05/03 09:55:57 UTC
kafka 0.9 offset unknown after cleanup
Hi,
I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up all old log segments(reach delete.retention time), I got unknown offset.
bin/kafka-consumer-groups.sh --bootstrap-server server:9092 --new-consumer --group testGroup --describe
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
In this situation, I cannot consume anything using new-consumer java driver if I disable auto-commit.
I think this happens because new-consumer driver stores offset in broker as a topic(not in zookeeper), and after reaching delete.retention time, it got deleted and becomes unknown. And since I disabled auto-commit, it can never know where it is, then it cannot consume anything.
Is this what happened here? What should I do in this situation?
Thanks,
Jun
Re: kafka 0.9 offset unknown after cleanup
Posted by Michael Freeman <mi...@gmail.com>.
Thanks Tom,
Thats a lot clearer now.
Michael
On Wed, May 4, 2016 at 3:27 PM, Tom Crayford <tc...@heroku.com> wrote:
> Michael,
>
> Increasing the offset retention period means that old consumers will use up
> space in the __consumer_offsets topic and in the cache the brokers hold on
> that topic. The cache is in memory, so if you churn consumer groups a lot
> that could be problematic. The other issue is that booting new brokers
> requires replaying the whole log to load the cache, and if it's larger then
> that could be problematic.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Wednesday, 4 May 2016, Michael Freeman <mi...@gmail.com> wrote:
>
> > Hey Tom,
> > Are there any details on the negative side effects of
> > increasing the offset retention period? I'd like to increase it but want
> to
> > be aware of the risks.
> >
> > Thanks
> >
> > Michael
> >
> > > On 4 May 2016, at 05:06, Tom Crayford <tcrayford@heroku.com
> > <javascript:;>> wrote:
> > >
> > > Jun,
> > >
> > > Yep, you got it. If there are no offset commits for 24h, then your
> offset
> > > will disappear and be cleaned up. I think this behaviour is at least a
> > > little suboptimal and surprising, but it does prevent a lot of issues
> > that
> > > arise when you rapidly add and remove consumer groups.
> > >
> > > I hope that helps!
> > >
> > > Tom Crayford,
> > > Heroku Kafka
> > >
> > >> On Tue, May 3, 2016 at 7:45 PM, Jun MA <mj.saber1990@gmail.com
> > <javascript:;>> wrote:
> > >>
> > >> I think I figured that out. Based on the explanation on
> > >> http://www.slideshare.net/jjkoshy/offset-management-in-kafka <
> > >> http://www.slideshare.net/jjkoshy/offset-management-in-kafka>,
> > >> offsets.retention.minutes is used for clean up dead consumer group. It
> > >> means if one consumer group hasn’t commit any offset for
> > >> offsets.retention.minutes, kafka will clean up its offset, which make
> > sense
> > >> for my case.
> > >>
> > >> Thanks,
> > >> Jun
> > >>> On May 3, 2016, at 11:46 AM, Jun MA <mj.saber1990@gmail.com
> > <javascript:;>> wrote:
> > >>>
> > >>> Thanks for your reply. I checked the offset topic and the cleanup
> > policy
> > >> is actually compact.
> > >>>
> > >>> Topic:__consumer_offsets PartitionCount:50
> > >> ReplicationFactor:3
> > >>
> >
> Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed
> > >>>
> > >>> And I’m using 0.9.0.1 so the default config for log.cleaner.enable is
> > >> true.
> > >>> In this case theoretically I should not lose any offset, right?
> > >>> I noticed that offsets.retention.minutes by default is 24hours, which
> > is
> > >> correlated to my offset becomes unknown, so I’m wondering if that is
> > >> because of this config? Can someone explain more about this
> > >> config(description: Log retention window for offsets topic) because I
> > don’t
> > >> understand what this does?
> > >>>
> > >>> Another related question is that how can I actually see the data of
> > >> offset topic? I think by reading the actual data might help.
> > >>>
> > >>> Thanks,
> > >>> Jun
> > >>>
> > >>>> On May 3, 2016, at 1:15 AM, Gerard Klijs <gerard.klijs@dizzit.com
> > <javascript:;>
> > >> <mailto:gerard.klijs@dizzit.com <javascript:;>>> wrote:
> > >>>>
> > >>>> Looks like it, you need to be sure the offset topic is using
> > compaction,
> > >>>> and the broker is set to enable compaction.
> > >>>>
> > >>>> On Tue, May 3, 2016 at 9:56 AM Jun MA <mj.saber1990@gmail.com
> > <javascript:;> <mailto:
> > >> mj.saber1990@gmail.com <javascript:;>>> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka
> cleans
> > >> up
> > >>>>> all old log segments(reach delete.retention time), I got unknown
> > >> offset.
> > >>>>>
> > >>>>> bin/kafka-consumer-groups.sh --bootstrap-server server:9092
> > >> --new-consumer
> > >>>>> --group testGroup --describe
> > >>>>> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> > >>>>> testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
> > >>>>> testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
> > >>>>>
> > >>>>> In this situation, I cannot consume anything using new-consumer
> java
> > >>>>> driver if I disable auto-commit.
> > >>>>> I think this happens because new-consumer driver stores offset in
> > >> broker
> > >>>>> as a topic(not in zookeeper), and after reaching delete.retention
> > >> time, it
> > >>>>> got deleted and becomes unknown. And since I disabled auto-commit,
> it
> > >> can
> > >>>>> never know where it is, then it cannot consume anything.
> > >>>>>
> > >>>>> Is this what happened here? What should I do in this situation?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Jun
> > >>
> > >>
> >
>
Re: kafka 0.9 offset unknown after cleanup
Posted by Tom Crayford <tc...@heroku.com>.
Michael,
Increasing the offset retention period means that old consumers will use up
space in the __consumer_offsets topic and in the cache the brokers hold on
that topic. The cache is in memory, so if you churn consumer groups a lot
that could be problematic. The other issue is that booting new brokers
requires replaying the whole log to load the cache, and if it's larger then
that could be problematic.
Thanks
Tom Crayford
Heroku Kafka
On Wednesday, 4 May 2016, Michael Freeman <mi...@gmail.com> wrote:
> Hey Tom,
> Are there any details on the negative side effects of
> increasing the offset retention period? I'd like to increase it but want to
> be aware of the risks.
>
> Thanks
>
> Michael
>
> > On 4 May 2016, at 05:06, Tom Crayford <tcrayford@heroku.com
> <javascript:;>> wrote:
> >
> > Jun,
> >
> > Yep, you got it. If there are no offset commits for 24h, then your offset
> > will disappear and be cleaned up. I think this behaviour is at least a
> > little suboptimal and surprising, but it does prevent a lot of issues
> that
> > arise when you rapidly add and remove consumer groups.
> >
> > I hope that helps!
> >
> > Tom Crayford,
> > Heroku Kafka
> >
> >> On Tue, May 3, 2016 at 7:45 PM, Jun MA <mj.saber1990@gmail.com
> <javascript:;>> wrote:
> >>
> >> I think I figured that out. Based on the explanation on
> >> http://www.slideshare.net/jjkoshy/offset-management-in-kafka <
> >> http://www.slideshare.net/jjkoshy/offset-management-in-kafka>,
> >> offsets.retention.minutes is used for clean up dead consumer group. It
> >> means if one consumer group hasn’t commit any offset for
> >> offsets.retention.minutes, kafka will clean up its offset, which make
> sense
> >> for my case.
> >>
> >> Thanks,
> >> Jun
> >>> On May 3, 2016, at 11:46 AM, Jun MA <mj.saber1990@gmail.com
> <javascript:;>> wrote:
> >>>
> >>> Thanks for your reply. I checked the offset topic and the cleanup
> policy
> >> is actually compact.
> >>>
> >>> Topic:__consumer_offsets PartitionCount:50
> >> ReplicationFactor:3
> >>
> Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed
> >>>
> >>> And I’m using 0.9.0.1 so the default config for log.cleaner.enable is
> >> true.
> >>> In this case theoretically I should not lose any offset, right?
> >>> I noticed that offsets.retention.minutes by default is 24hours, which
> is
> >> correlated to my offset becomes unknown, so I’m wondering if that is
> >> because of this config? Can someone explain more about this
> >> config(description: Log retention window for offsets topic) because I
> don’t
> >> understand what this does?
> >>>
> >>> Another related question is that how can I actually see the data of
> >> offset topic? I think by reading the actual data might help.
> >>>
> >>> Thanks,
> >>> Jun
> >>>
> >>>> On May 3, 2016, at 1:15 AM, Gerard Klijs <gerard.klijs@dizzit.com
> <javascript:;>
> >> <mailto:gerard.klijs@dizzit.com <javascript:;>>> wrote:
> >>>>
> >>>> Looks like it, you need to be sure the offset topic is using
> compaction,
> >>>> and the broker is set to enable compaction.
> >>>>
> >>>> On Tue, May 3, 2016 at 9:56 AM Jun MA <mj.saber1990@gmail.com
> <javascript:;> <mailto:
> >> mj.saber1990@gmail.com <javascript:;>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans
> >> up
> >>>>> all old log segments(reach delete.retention time), I got unknown
> >> offset.
> >>>>>
> >>>>> bin/kafka-consumer-groups.sh --bootstrap-server server:9092
> >> --new-consumer
> >>>>> --group testGroup --describe
> >>>>> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> >>>>> testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
> >>>>> testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
> >>>>>
> >>>>> In this situation, I cannot consume anything using new-consumer java
> >>>>> driver if I disable auto-commit.
> >>>>> I think this happens because new-consumer driver stores offset in
> >> broker
> >>>>> as a topic(not in zookeeper), and after reaching delete.retention
> >> time, it
> >>>>> got deleted and becomes unknown. And since I disabled auto-commit, it
> >> can
> >>>>> never know where it is, then it cannot consume anything.
> >>>>>
> >>>>> Is this what happened here? What should I do in this situation?
> >>>>>
> >>>>> Thanks,
> >>>>> Jun
> >>
> >>
>
Re: kafka 0.9 offset unknown after cleanup
Posted by Michael Freeman <mi...@gmail.com>.
Hey Tom,
Are there any details on the negative side effects of increasing the offset retention period? I'd like to increase it but want to be aware of the risks.
Thanks
Michael
> On 4 May 2016, at 05:06, Tom Crayford <tc...@heroku.com> wrote:
>
> Jun,
>
> Yep, you got it. If there are no offset commits for 24h, then your offset
> will disappear and be cleaned up. I think this behaviour is at least a
> little suboptimal and surprising, but it does prevent a lot of issues that
> arise when you rapidly add and remove consumer groups.
>
> I hope that helps!
>
> Tom Crayford,
> Heroku Kafka
>
>> On Tue, May 3, 2016 at 7:45 PM, Jun MA <mj...@gmail.com> wrote:
>>
>> I think I figured that out. Based on the explanation on
>> http://www.slideshare.net/jjkoshy/offset-management-in-kafka <
>> http://www.slideshare.net/jjkoshy/offset-management-in-kafka>,
>> offsets.retention.minutes is used for clean up dead consumer group. It
>> means if one consumer group hasn’t commit any offset for
>> offsets.retention.minutes, kafka will clean up its offset, which make sense
>> for my case.
>>
>> Thanks,
>> Jun
>>> On May 3, 2016, at 11:46 AM, Jun MA <mj...@gmail.com> wrote:
>>>
>>> Thanks for your reply. I checked the offset topic and the cleanup policy
>> is actually compact.
>>>
>>> Topic:__consumer_offsets PartitionCount:50
>> ReplicationFactor:3
>> Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed
>>>
>>> And I’m using 0.9.0.1 so the default config for log.cleaner.enable is
>> true.
>>> In this case theoretically I should not lose any offset, right?
>>> I noticed that offsets.retention.minutes by default is 24hours, which is
>> correlated to my offset becomes unknown, so I’m wondering if that is
>> because of this config? Can someone explain more about this
>> config(description: Log retention window for offsets topic) because I don’t
>> understand what this does?
>>>
>>> Another related question is that how can I actually see the data of
>> offset topic? I think by reading the actual data might help.
>>>
>>> Thanks,
>>> Jun
>>>
>>>> On May 3, 2016, at 1:15 AM, Gerard Klijs <gerard.klijs@dizzit.com
>> <ma...@dizzit.com>> wrote:
>>>>
>>>> Looks like it, you need to be sure the offset topic is using compaction,
>>>> and the broker is set to enable compaction.
>>>>
>>>> On Tue, May 3, 2016 at 9:56 AM Jun MA <mj.saber1990@gmail.com <mailto:
>> mj.saber1990@gmail.com>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans
>> up
>>>>> all old log segments(reach delete.retention time), I got unknown
>> offset.
>>>>>
>>>>> bin/kafka-consumer-groups.sh --bootstrap-server server:9092
>> --new-consumer
>>>>> --group testGroup --describe
>>>>> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
>>>>> testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
>>>>> testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
>>>>>
>>>>> In this situation, I cannot consume anything using new-consumer java
>>>>> driver if I disable auto-commit.
>>>>> I think this happens because new-consumer driver stores offset in
>> broker
>>>>> as a topic(not in zookeeper), and after reaching delete.retention
>> time, it
>>>>> got deleted and becomes unknown. And since I disabled auto-commit, it
>> can
>>>>> never know where it is, then it cannot consume anything.
>>>>>
>>>>> Is this what happened here? What should I do in this situation?
>>>>>
>>>>> Thanks,
>>>>> Jun
>>
>>
Re: kafka 0.9 offset unknown after cleanup
Posted by Tom Crayford <tc...@heroku.com>.
Jun,
Yep, you got it. If there are no offset commits for 24h, then your offset
will disappear and be cleaned up. I think this behaviour is at least a
little suboptimal and surprising, but it does prevent a lot of issues that
arise when you rapidly add and remove consumer groups.
I hope that helps!
Tom Crayford,
Heroku Kafka
On Tue, May 3, 2016 at 7:45 PM, Jun MA <mj...@gmail.com> wrote:
> I think I figured that out. Based on the explanation on
> http://www.slideshare.net/jjkoshy/offset-management-in-kafka <
> http://www.slideshare.net/jjkoshy/offset-management-in-kafka>,
> offsets.retention.minutes is used for clean up dead consumer group. It
> means if one consumer group hasn’t commit any offset for
> offsets.retention.minutes, kafka will clean up its offset, which make sense
> for my case.
>
> Thanks,
> Jun
> > On May 3, 2016, at 11:46 AM, Jun MA <mj...@gmail.com> wrote:
> >
> > Thanks for your reply. I checked the offset topic and the cleanup policy
> is actually compact.
> >
> > Topic:__consumer_offsets PartitionCount:50
> ReplicationFactor:3
> Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed
> >
> > And I’m using 0.9.0.1 so the default config for log.cleaner.enable is
> true.
> > In this case theoretically I should not lose any offset, right?
> > I noticed that offsets.retention.minutes by default is 24hours, which is
> correlated to my offset becomes unknown, so I’m wondering if that is
> because of this config? Can someone explain more about this
> config(description: Log retention window for offsets topic) because I don’t
> understand what this does?
> >
> > Another related question is that how can I actually see the data of
> offset topic? I think by reading the actual data might help.
> >
> > Thanks,
> > Jun
> >
> >> On May 3, 2016, at 1:15 AM, Gerard Klijs <gerard.klijs@dizzit.com
> <ma...@dizzit.com>> wrote:
> >>
> >> Looks like it, you need to be sure the offset topic is using compaction,
> >> and the broker is set to enable compaction.
> >>
> >> On Tue, May 3, 2016 at 9:56 AM Jun MA <mj.saber1990@gmail.com <mailto:
> mj.saber1990@gmail.com>> wrote:
> >>
> >>> Hi,
> >>>
> >>> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans
> up
> >>> all old log segments(reach delete.retention time), I got unknown
> offset.
> >>>
> >>> bin/kafka-consumer-groups.sh --bootstrap-server server:9092
> --new-consumer
> >>> --group testGroup --describe
> >>> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> >>> testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
> >>> testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
> >>>
> >>> In this situation, I cannot consume anything using new-consumer java
> >>> driver if I disable auto-commit.
> >>> I think this happens because new-consumer driver stores offset in
> broker
> >>> as a topic(not in zookeeper), and after reaching delete.retention
> time, it
> >>> got deleted and becomes unknown. And since I disabled auto-commit, it
> can
> >>> never know where it is, then it cannot consume anything.
> >>>
> >>> Is this what happened here? What should I do in this situation?
> >>>
> >>> Thanks,
> >>> Jun
> >
>
>
Re: kafka 0.9 offset unknown after cleanup
Posted by Jun MA <mj...@gmail.com>.
I think I figured that out. Based on the explanation on http://www.slideshare.net/jjkoshy/offset-management-in-kafka <http://www.slideshare.net/jjkoshy/offset-management-in-kafka>, offsets.retention.minutes is used for clean up dead consumer group. It means if one consumer group hasn’t commit any offset for offsets.retention.minutes, kafka will clean up its offset, which make sense for my case.
Thanks,
Jun
> On May 3, 2016, at 11:46 AM, Jun MA <mj...@gmail.com> wrote:
>
> Thanks for your reply. I checked the offset topic and the cleanup policy is actually compact.
>
> Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed
>
> And I’m using 0.9.0.1 so the default config for log.cleaner.enable is true.
> In this case theoretically I should not lose any offset, right?
> I noticed that offsets.retention.minutes by default is 24hours, which is correlated to my offset becomes unknown, so I’m wondering if that is because of this config? Can someone explain more about this config(description: Log retention window for offsets topic) because I don’t understand what this does?
>
> Another related question is that how can I actually see the data of offset topic? I think by reading the actual data might help.
>
> Thanks,
> Jun
>
>> On May 3, 2016, at 1:15 AM, Gerard Klijs <gerard.klijs@dizzit.com <ma...@dizzit.com>> wrote:
>>
>> Looks like it, you need to be sure the offset topic is using compaction,
>> and the broker is set to enable compaction.
>>
>> On Tue, May 3, 2016 at 9:56 AM Jun MA <mj.saber1990@gmail.com <ma...@gmail.com>> wrote:
>>
>>> Hi,
>>>
>>> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up
>>> all old log segments(reach delete.retention time), I got unknown offset.
>>>
>>> bin/kafka-consumer-groups.sh --bootstrap-server server:9092 --new-consumer
>>> --group testGroup --describe
>>> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
>>> testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
>>> testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
>>>
>>> In this situation, I cannot consume anything using new-consumer java
>>> driver if I disable auto-commit.
>>> I think this happens because new-consumer driver stores offset in broker
>>> as a topic(not in zookeeper), and after reaching delete.retention time, it
>>> got deleted and becomes unknown. And since I disabled auto-commit, it can
>>> never know where it is, then it cannot consume anything.
>>>
>>> Is this what happened here? What should I do in this situation?
>>>
>>> Thanks,
>>> Jun
>
Re: kafka 0.9 offset unknown after cleanup
Posted by Jun MA <mj...@gmail.com>.
Thanks for your reply. I checked the offset topic and the cleanup policy is actually compact.
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed
And I’m using 0.9.0.1 so the default config for log.cleaner.enable is true.
In this case theoretically I should not lose any offset, right?
I noticed that offsets.retention.minutes by default is 24hours, which is correlated to my offset becomes unknown, so I’m wondering if that is because of this config? Can someone explain more about this config(description: Log retention window for offsets topic) because I don’t understand what this does?
Another related question is that how can I actually see the data of offset topic? I think by reading the actual data might help.
Thanks,
Jun
> On May 3, 2016, at 1:15 AM, Gerard Klijs <ge...@dizzit.com> wrote:
>
> Looks like it, you need to be sure the offset topic is using compaction,
> and the broker is set to enable compaction.
>
> On Tue, May 3, 2016 at 9:56 AM Jun MA <mj...@gmail.com> wrote:
>
>> Hi,
>>
>> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up
>> all old log segments(reach delete.retention time), I got unknown offset.
>>
>> bin/kafka-consumer-groups.sh --bootstrap-server server:9092 --new-consumer
>> --group testGroup --describe
>> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
>> testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
>> testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
>>
>> In this situation, I cannot consume anything using new-consumer java
>> driver if I disable auto-commit.
>> I think this happens because new-consumer driver stores offset in broker
>> as a topic(not in zookeeper), and after reaching delete.retention time, it
>> got deleted and becomes unknown. And since I disabled auto-commit, it can
>> never know where it is, then it cannot consume anything.
>>
>> Is this what happened here? What should I do in this situation?
>>
>> Thanks,
>> Jun
Re: kafka 0.9 offset unknown after cleanup
Posted by Gerard Klijs <ge...@dizzit.com>.
Looks like it, you need to be sure the offset topic is using compaction,
and the broker is set to enable compaction.
On Tue, May 3, 2016 at 9:56 AM Jun MA <mj...@gmail.com> wrote:
> Hi,
>
> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up
> all old log segments(reach delete.retention time), I got unknown offset.
>
> bin/kafka-consumer-groups.sh --bootstrap-server server:9092 --new-consumer
> --group testGroup --describe
> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
> testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
>
> In this situation, I cannot consume anything using new-consumer java
> driver if I disable auto-commit.
> I think this happens because new-consumer driver stores offset in broker
> as a topic(not in zookeeper), and after reaching delete.retention time, it
> got deleted and becomes unknown. And since I disabled auto-commit, it can
> never know where it is, then it cannot consume anything.
>
> Is this what happened here? What should I do in this situation?
>
> Thanks,
> Jun