You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Bateman, Matt" <ma...@ebay.com> on 2012/03/24 02:09:32 UTC
Log Cleaner in Continuous Loop?
Has anyone else see the cleaner get into a continuous loop, attempting to delete the same log files over and over for topic partitions? We've hit this twice now where it never stops until we bounce the broker.
I didn't see a JIRA or a mention in the mailing list archives.
Thanks,
Matt
RE: Log Cleaner in Continuous Loop?
Posted by "Bateman, Matt" <ma...@ebay.com>.
Hi Jun,
Sounds like it. These topics have a very low frequency of messages.
Thanks,
Matt
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Friday, March 23, 2012 10:46 PM
To: kafka-users@incubator.apache.org
Subject: Re: Log Cleaner in Continuous Loop?
https://issues.apache.org/jira/browse/KAFKA-292 can cause this when a topic gets new data infrequently. Is that the same problem that you have seen?
Thanks,
Jun
On Fri, Mar 23, 2012 at 6:09 PM, Bateman, Matt <ma...@ebay.com> wrote:
> Has anyone else see the cleaner get into a continuous loop, attempting
> to delete the same log files over and over for topic partitions? We've
> hit this twice now where it never stops until we bounce the broker.
>
> I didn't see a JIRA or a mention in the mailing list archives.
>
> Thanks,
>
> Matt
>
>
Re: Log Cleaner in Continuous Loop?
Posted by Chris Burroughs <ch...@gmail.com>.
Thanks!
On 03/25/2012 11:50 PM, Jun Rao wrote:
> Yes. This bug causes the last log segment file (if empty) to be
> deleted repeatedly every time the log cleaner thread wakes up (by default,
> every minute).
>
> Jun
>
> On Sun, Mar 25, 2012 at 7:17 PM, Chris Burroughs
> <ch...@gmail.com>wrote:
>
>> Jun, will KAFKA-292 result in output like this?
>>
>> $ grep t00-0 log/kafka.log | head
>> 2012-03-25 19:42:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>> 2012-03-25 19:43:03,941 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>> 2012-03-25 19:44:03,942 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>> 2012-03-25 19:45:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>> 2012-03-25 19:46:03,942 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>> 2012-03-25 19:47:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>> 2012-03-25 19:48:03,942 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>> 2012-03-25 19:49:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>> 2012-03-25 19:50:03,941 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>> 2012-03-25 19:51:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
>> Deleting log segment 00000000000000000000.kafka from t00-0
>>
>>
>> On 03/24/2012 01:45 AM, Jun Rao wrote:
>>> https://issues.apache.org/jira/browse/KAFKA-292 can cause this when a
>> topic
>>> gets new data infrequently. Is that the same problem that you have seen?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Fri, Mar 23, 2012 at 6:09 PM, Bateman, Matt <ma...@ebay.com>
>> wrote:
>>>
>>>> Has anyone else see the cleaner get into a continuous loop, attempting
>> to
>>>> delete the same log files over and over for topic partitions? We've hit
>>>> this twice now where it never stops until we bounce the broker.
>>>>
>>>> I didn't see a JIRA or a mention in the mailing list archives.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>
>>
>>
>
Re: Log Cleaner in Continuous Loop?
Posted by Chris Burroughs <ch...@gmail.com>.
Jun, will KAFKA-292 result in output like this?
$ grep t00-0 log/kafka.log | head
2012-03-25 19:42:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
2012-03-25 19:43:03,941 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
2012-03-25 19:44:03,942 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
2012-03-25 19:45:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
2012-03-25 19:46:03,942 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
2012-03-25 19:47:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
2012-03-25 19:48:03,942 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
2012-03-25 19:49:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
2012-03-25 19:50:03,941 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
2012-03-25 19:51:03,943 LogManager.scala:201 kafka-logcleaner-0 INFO |
Deleting log segment 00000000000000000000.kafka from t00-0
On 03/24/2012 01:45 AM, Jun Rao wrote:
> https://issues.apache.org/jira/browse/KAFKA-292 can cause this when a topic
> gets new data infrequently. Is that the same problem that you have seen?
>
> Thanks,
>
> Jun
>
> On Fri, Mar 23, 2012 at 6:09 PM, Bateman, Matt <ma...@ebay.com> wrote:
>
>> Has anyone else see the cleaner get into a continuous loop, attempting to
>> delete the same log files over and over for topic partitions? We've hit
>> this twice now where it never stops until we bounce the broker.
>>
>> I didn't see a JIRA or a mention in the mailing list archives.
>>
>> Thanks,
>>
>> Matt
>>
>>
>
Re: Log Cleaner in Continuous Loop?
Posted by Jun Rao <ju...@gmail.com>.
https://issues.apache.org/jira/browse/KAFKA-292 can cause this when a topic
gets new data infrequently. Is that the same problem that you have seen?
Thanks,
Jun
On Fri, Mar 23, 2012 at 6:09 PM, Bateman, Matt <ma...@ebay.com> wrote:
> Has anyone else see the cleaner get into a continuous loop, attempting to
> delete the same log files over and over for topic partitions? We've hit
> this twice now where it never stops until we bounce the broker.
>
> I didn't see a JIRA or a mention in the mailing list archives.
>
> Thanks,
>
> Matt
>
>