Can I delete old log files manually?

[Tom Amon <ta...@gmail.com>]

My apologies for mail bombing the list. I'm banging my head against a
production issue.

In short, can I delete old log and index files manually? I have two brokers
out of five that are hosting ~1200 partitions each, though based on my
understanding from a previous email they should really only be hosting
about 700 and are sized as such. I'm running out of disk space rapidly. I
have my purge set to 3 days and there are logs in there that are 3 days old
but haven't been purged yet. I'm worried I'm going to run out of disk space
before they are deleted.

Is this safe?

I think I need to specify the size as well to help mitigate this but
without knowing how many partitions will be hosted it's tough to size that
as well.

Thanks for your patience.

Re: Can I delete old log files manually?

[Neha Narkhede <ne...@gmail.com>]

 I have my purge set to 3 days and there are logs in there that are 3 days
old
but haven't been purged yet.

Older log segments are purged only when the log segment rolls over. This is
controlled by the log.segment.bytes config. You might want to lower this in
your cluster to let Kafka purge the segments relatively sooner. However,
note that the side effect of doing so is increase in the # of open file
handles. You might have to bump up the allowable # of open file handles per
broker.

Regarding your original question, yes you can delete older log and index
files (provided you leave around the latest log and index file). Once you
delete these, you will have to bounce the broker so it can clean the file
handles from it's internal data structures.

Hope that helps!
-Neha


On Tue, Mar 25, 2014 at 11:52 AM, Tom Amon <ta...@gmail.com> wrote:

> My apologies for mail bombing the list. I'm banging my head against a
> production issue.
>
> In short, can I delete old log and index files manually? I have two brokers
> out of five that are hosting ~1200 partitions each, though based on my
> understanding from a previous email they should really only be hosting
> about 700 and are sized as such. I'm running out of disk space rapidly. I
> have my purge set to 3 days and there are logs in there that are 3 days old
> but haven't been purged yet. I'm worried I'm going to run out of disk space
> before they are deleted.
>
> Is this safe?
>
> I think I need to specify the size as well to help mitigate this but
> without knowing how many partitions will be hosted it's tough to size that
> as well.
>
> Thanks for your patience.
>