You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jörg Wagner <jo...@1und1.de> on 2016/05/06 12:12:24 UTC
unknown (kafka) offsets after restart
We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
they are stored in kafka topics.
Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
happen that the offsets are unknown and therefore stop consumers from
consuming without knowing their offset.
kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group
--zookeeper localhost:2181
Could not fetch offset for [topic,19] due to
kafka.common.OffsetsLoadInProgressException.
Group Topic Pid Offset
logSize Lag Owner
group topic 0 unknown 326606 unknown none
I currently have no other solution to this than to wait.. but it takes a
very long time (hours.. the servers are hopelessly underutilized!), I
would be grateful for any advice.
Thanks
J�rg
Re: unknown (kafka) offsets after restart
Posted by Jörg Wagner <jo...@1und1.de>.
I dug further into this and log.cleaner.enable is false by default in
0.8.2.x which pretty much explains all this. Fixing it now.
Thanks!
On 09.05.2016 08:17, J�rg Wagner wrote:
> Thank you James.
>
> The log cleaner log is empty and we didn't notice any issue there. The
> Topic however is ~174GB per Node (3) which would support your
> information about the log cleaner dying.
>
> Since this is a production system I do not think debug logging is
> viable even for a short time. Could you maybe point me in the
> direction of debugging compaction where it went wrong?
>
> Thanks
> J�rg
>
> On 06.05.2016 22:21, James Cheng wrote:
>> Is the log compaction thread correctly working? The offsets are
>> stored in a log compacted topic, and we have seen issues where the
>> log cleaner thread dies and therefore the offsets topic just grows
>> forever, which means it will take a long time to read in the topic.
>>
>> You can look in the log-cleaner.log debuglog file to see if there are
>> any error messages there.
>>
>> -James
>>
>>
>>> On May 6, 2016, at 6:28 AM, J�rg Wagner <jo...@1und1.de> wrote:
>>>
>>> After a bit more looking into this we found out that the
>>> Offsetmanager is single threaded and due to our setup (few, powerful
>>> servers: rather bad for kafka I know..) it seems we are limiting
>>> ourselves severely by using kafka offsets.
>>>
>>> Any more insight is still welcome of course.
>>>
>>>
>>> -------- Forwarded Message --------
>>> Subject: unknown (kafka) offsets after restart
>>> Date: Fri, 6 May 2016 14:12:24 +0200
>>> From: J�rg Wagner <jo...@1und1.de>
>>> Reply-To: users@kafka.apache.org
>>> To: users@kafka.apache.org
>>>
>>>
>>>
>>> We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
>>> they are stored in kafka topics.
>>>
>>> Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
>>> happen that the offsets are unknown and therefore stop consumers from
>>> consuming without knowing their offset.
>>>
>>> kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group
>>> --zookeeper localhost:2181
>>> Could not fetch offset for [topic,19] due to
>>> kafka.common.OffsetsLoadInProgressException.
>>>
>>> Group Topic Pid Offset
>>> logSize Lag Owner
>>> group topic 0 unknown 326606 unknown none
>>>
>>> I currently have no other solution to this than to wait.. but it
>>> takes a
>>> very long time (hours.. the servers are hopelessly underutilized!), I
>>> would be grateful for any advice.
>>>
>>> Thanks
>>> J�rg
>>>
>>>
>>>
>
--
Mit freundlichem Gru�
J�rg Wagner
Systemadministrator
Search & Account Security
1&1 Mail & Media Development & Technology GmbH | Sapporobogen 6-8 | 80637 M�nchen | Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagner1@1und1.de | Web: www.1und1.de
Amtsgericht Montabaur, HRB 5452
Gesch�ftsf�hrer: Frank Einhellinger, Thomas Ludwig, Jan Oetjen
Member of United Internet
Re: unknown (kafka) offsets after restart
Posted by Jörg Wagner <jo...@1und1.de>.
Thank you James.
The log cleaner log is empty and we didn't notice any issue there. The
Topic however is ~174GB per Node (3) which would support your
information about the log cleaner dying.
Since this is a production system I do not think debug logging is viable
even for a short time. Could you maybe point me in the direction of
debugging compaction where it went wrong?
Thanks
J�rg
On 06.05.2016 22:21, James Cheng wrote:
> Is the log compaction thread correctly working? The offsets are stored in a log compacted topic, and we have seen issues where the log cleaner thread dies and therefore the offsets topic just grows forever, which means it will take a long time to read in the topic.
>
> You can look in the log-cleaner.log debuglog file to see if there are any error messages there.
>
> -James
>
>
>> On May 6, 2016, at 6:28 AM, J�rg Wagner <jo...@1und1.de> wrote:
>>
>> After a bit more looking into this we found out that the Offsetmanager is single threaded and due to our setup (few, powerful servers: rather bad for kafka I know..) it seems we are limiting ourselves severely by using kafka offsets.
>>
>> Any more insight is still welcome of course.
>>
>>
>> -------- Forwarded Message --------
>> Subject: unknown (kafka) offsets after restart
>> Date: Fri, 6 May 2016 14:12:24 +0200
>> From: J�rg Wagner <jo...@1und1.de>
>> Reply-To: users@kafka.apache.org
>> To: users@kafka.apache.org
>>
>>
>>
>> We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
>> they are stored in kafka topics.
>>
>> Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
>> happen that the offsets are unknown and therefore stop consumers from
>> consuming without knowing their offset.
>>
>> kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group
>> --zookeeper localhost:2181
>> Could not fetch offset for [topic,19] due to
>> kafka.common.OffsetsLoadInProgressException.
>>
>> Group Topic Pid Offset
>> logSize Lag Owner
>> group topic 0 unknown 326606 unknown none
>>
>> I currently have no other solution to this than to wait.. but it takes a
>> very long time (hours.. the servers are hopelessly underutilized!), I
>> would be grateful for any advice.
>>
>> Thanks
>> J�rg
>>
>>
>>
--
Mit freundlichem Gru�
J�rg Wagner
Systemadministrator
Search & Account Security
1&1 Mail & Media Development & Technology GmbH | Sapporobogen 6-8 | 80637 M�nchen | Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagner1@1und1.de | Web: www.1und1.de
Amtsgericht Montabaur, HRB 5452
Gesch�ftsf�hrer: Frank Einhellinger, Thomas Ludwig, Jan Oetjen
Member of United Internet
Re: unknown (kafka) offsets after restart
Posted by James Cheng <wu...@gmail.com>.
Is the log compaction thread correctly working? The offsets are stored in a log compacted topic, and we have seen issues where the log cleaner thread dies and therefore the offsets topic just grows forever, which means it will take a long time to read in the topic.
You can look in the log-cleaner.log debuglog file to see if there are any error messages there.
-James
> On May 6, 2016, at 6:28 AM, Jörg Wagner <jo...@1und1.de> wrote:
>
> After a bit more looking into this we found out that the Offsetmanager is single threaded and due to our setup (few, powerful servers: rather bad for kafka I know..) it seems we are limiting ourselves severely by using kafka offsets.
>
> Any more insight is still welcome of course.
>
>
> -------- Forwarded Message --------
> Subject: unknown (kafka) offsets after restart
> Date: Fri, 6 May 2016 14:12:24 +0200
> From: Jörg Wagner <jo...@1und1.de>
> Reply-To: users@kafka.apache.org
> To: users@kafka.apache.org
>
>
>
> We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
> they are stored in kafka topics.
>
> Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
> happen that the offsets are unknown and therefore stop consumers from
> consuming without knowing their offset.
>
> kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group
> --zookeeper localhost:2181
> Could not fetch offset for [topic,19] due to
> kafka.common.OffsetsLoadInProgressException.
>
> Group Topic Pid Offset
> logSize Lag Owner
> group topic 0 unknown 326606 unknown none
>
> I currently have no other solution to this than to wait.. but it takes a
> very long time (hours.. the servers are hopelessly underutilized!), I
> would be grateful for any advice.
>
> Thanks
> Jörg
>
>
>
Fwd: unknown (kafka) offsets after restart
Posted by Jörg Wagner <jo...@1und1.de>.
After a bit more looking into this we found out that the Offsetmanager
is single threaded and due to our setup (few, powerful servers: rather
bad for kafka I know..) it seems we are limiting ourselves severely by
using kafka offsets.
Any more insight is still welcome of course.
-------- Forwarded Message --------
Subject: unknown (kafka) offsets after restart
Date: Fri, 6 May 2016 14:12:24 +0200
From: J�rg Wagner <jo...@1und1.de>
Reply-To: users@kafka.apache.org
To: users@kafka.apache.org
We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
they are stored in kafka topics.
Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
happen that the offsets are unknown and therefore stop consumers from
consuming without knowing their offset.
kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group
--zookeeper localhost:2181
Could not fetch offset for [topic,19] due to
kafka.common.OffsetsLoadInProgressException.
Group Topic Pid Offset
logSize Lag Owner
group topic 0 unknown 326606 unknown none
I currently have no other solution to this than to wait.. but it takes a
very long time (hours.. the servers are hopelessly underutilized!), I
would be grateful for any advice.
Thanks
J�rg