You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dhirendra Singh <dh...@gmail.com> on 2021/11/15 05:30:35 UTC
Log cleaner is marking the partition as uncleanable
Hi All,
I have a kafka cluster with 3 brokers running in kubernetes. kafka version
i am using is 2.7.1
Frequently we are observing that log cleaner is not able to comapct and
clean log of some partitions. it mark the partition as uncleanable.
Following is the error message we see in the log...
[2021-11-14 23:04:32,215] WARN [kafka-log-cleaner-thread-0]: Unexpected
exception thrown when cleaning log Log(dir=/home/kafka/data/order-0,
topic=order, partition=0, highWatermark=50158328,
lastStableOffset=50158328, logStartOffset=0, logEndOffset=50158328).
Marking its partition (order-0) as uncleanable (kafka.log.LogCleaner)
kafka.log.LogCleaningException: Record size 0 is less than the minimum
record overhead (14)
at
kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:357)
at
kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:333)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:322)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Caused by: org.apache.kafka.common.errors.CorruptRecordException: Record
size 0 is less than the minimum record overhead (14)
If i delete the topic and recrete it partition remain uncleanable unless
brokers are restarted after deleteing and recreating the topic.
what is causing this issue and how to fix it ?
Thanks,
Dhirendra.
Re: Log cleaner is marking the partition as uncleanable
Posted by Dhirendra Singh <dh...@gmail.com>.
Hi Luke,
What could be the reason for log corruption ?
There is no abnormal broker shutdown or the machine where broker is running
has any failure.
we see this error in the log while broker is up.
Moreover when i delete the topic and recreate it log cleaner do not remove
the topic from uncleanable list and log size continue to grow.
Thanks,
Dhirendra.
On Mon, Nov 15, 2021 at 2:40 PM Luke Chen <sh...@gmail.com> wrote:
> Hi Dhirendra,
> The error message said that the log cleaner believed your log file is
> corrupted because there's a record with 0 Byte, but the minimum record size
> is 14 Bytes (including CRC, Key, Value,...). I don't think it's good to fix
> the corrupted file manually.
>
> Thank you.
> Luke
>
> On Mon, Nov 15, 2021 at 1:31 PM Dhirendra Singh <dh...@gmail.com>
> wrote:
>
> > Hi All,
> > I have a kafka cluster with 3 brokers running in kubernetes. kafka
> version
> > i am using is 2.7.1
> > Frequently we are observing that log cleaner is not able to comapct and
> > clean log of some partitions. it mark the partition as uncleanable.
> > Following is the error message we see in the log...
> >
> > [2021-11-14 23:04:32,215] WARN [kafka-log-cleaner-thread-0]: Unexpected
> > exception thrown when cleaning log Log(dir=/home/kafka/data/order-0,
> > topic=order, partition=0, highWatermark=50158328,
> > lastStableOffset=50158328, logStartOffset=0, logEndOffset=50158328).
> > Marking its partition (order-0) as uncleanable (kafka.log.LogCleaner)
> > kafka.log.LogCleaningException: Record size 0 is less than the minimum
> > record overhead (14)
> > at
> >
> kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:357)
> > at
> >
> >
> kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:333)
> > at
> kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:322)
> > at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
> > Caused by: org.apache.kafka.common.errors.CorruptRecordException: Record
> > size 0 is less than the minimum record overhead (14)
> >
> > If i delete the topic and recrete it partition remain uncleanable unless
> > brokers are restarted after deleteing and recreating the topic.
> >
> > what is causing this issue and how to fix it ?
> >
> > Thanks,
> > Dhirendra.
> >
>
Re: Log cleaner is marking the partition as uncleanable
Posted by Luke Chen <sh...@gmail.com>.
Hi Dhirendra,
The error message said that the log cleaner believed your log file is
corrupted because there's a record with 0 Byte, but the minimum record size
is 14 Bytes (including CRC, Key, Value,...). I don't think it's good to fix
the corrupted file manually.
Thank you.
Luke
On Mon, Nov 15, 2021 at 1:31 PM Dhirendra Singh <dh...@gmail.com>
wrote:
> Hi All,
> I have a kafka cluster with 3 brokers running in kubernetes. kafka version
> i am using is 2.7.1
> Frequently we are observing that log cleaner is not able to comapct and
> clean log of some partitions. it mark the partition as uncleanable.
> Following is the error message we see in the log...
>
> [2021-11-14 23:04:32,215] WARN [kafka-log-cleaner-thread-0]: Unexpected
> exception thrown when cleaning log Log(dir=/home/kafka/data/order-0,
> topic=order, partition=0, highWatermark=50158328,
> lastStableOffset=50158328, logStartOffset=0, logEndOffset=50158328).
> Marking its partition (order-0) as uncleanable (kafka.log.LogCleaner)
> kafka.log.LogCleaningException: Record size 0 is less than the minimum
> record overhead (14)
> at
> kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:357)
> at
>
> kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:333)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:322)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
> Caused by: org.apache.kafka.common.errors.CorruptRecordException: Record
> size 0 is less than the minimum record overhead (14)
>
> If i delete the topic and recrete it partition remain uncleanable unless
> brokers are restarted after deleteing and recreating the topic.
>
> what is causing this issue and how to fix it ?
>
> Thanks,
> Dhirendra.
>