You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vishnu Viswanath <vi...@gmail.com> on 2018/10/18 20:26:55 UTC

Strange log delete behavior after segment rollover

Hi All,

I am seeing some strange behavior on how the rolled over segments are
deleted. Here is the log for one of the partition.

[2018-10-18 07:49:31,557] Rolled new log segment at offset 3600 in 24 ms.
(kafka.log.Log)
[2018-10-18 07:51:32,443] Deleting segment 0 (kafka.log.Log)
^ Log was rolled over and deleted within 2 minutes. Which is good.

[2018-10-18 08:22:32,131] Rolled new log segment at offset 7216 in 261 ms.
(kafka.log.Log)
[2018-10-18 08:31:44,928] Deleting segment 0 (kafka.log.Log)
[2018-10-18 08:31:44,928] Deleting segment 3600 (kafka.log.Log)
^ Log was rolled over and deleted within 2 minutes. But, why kafka trying
to delete segment 0 again?


[2018-10-18 08:55:10,442] Rolled new log segment at offset 10816 in 1 ms.
(kafka.log.Log)
[2018-10-18 09:27:45,567] Rolled new log segment at offset 14420 in 1 ms.
(kafka.log.Log)
[2018-10-18 09:37:07,728] Deleting segment 0 (kafka.log.Log)
[2018-10-18 09:37:11,549] Deleting segment 7216 (kafka.log.Log)
[2018-10-18 09:37:14,356] Deleting segment 10816 (kafka.log.Log)
^ Again same behavior, but the time between rollover and deletion has
increased to 30 minutes.


[2018-10-18 10:00:37,208] Rolled new log segment at offset 18016 in 1 ms.
(kafka.log.Log)
[2018-10-18 10:32:43,677] Rolled new log segment at offset 21583 in 6 ms.
(kafka.log.Log)
[2018-10-18 10:35:28,808] Deleting segment 0 (kafka.log.Log)
[2018-10-18 10:35:28,808] Deleting segment 7216 (kafka.log.Log)
[2018-10-18 10:35:28,809] Deleting segment 10816 (kafka.log.Log)
[2018-10-18 10:35:32,035] Deleting segment 14420 (kafka.log.Log)
[2018-10-18 10:35:33,953] Deleting segment 18016 (kafka.log.Log)
^ Again deletion is happening after about 35 mins.


[2018-10-18 11:05:18,246] Rolled new log segment at offset 25178 in 12 ms.
(kafka.log.Log)
[2018-10-18 11:37:55,385] Rolled new log segment at offset 28754 in 8 ms.
(kafka.log.Log)
[2018-10-18 12:10:20,695] Rolled new log segment at offset 32347 in 25 ms.
(kafka.log.Log)
[2018-10-18 12:43:07,535] Rolled new log segment at offset 35913 in 1 ms.
(kafka.log.Log)
[2018-10-18 12:57:10,200] Deleting segment 0 (kafka.log.Log)
[2018-10-18 12:57:10,200] Deleting segment 14420 (kafka.log.Log)
[2018-10-18 12:57:10,201] Deleting segment 18016 (kafka.log.Log)
[2018-10-18 12:57:12,584] Deleting segment 21583 (kafka.log.Log)
[2018-10-18 12:57:16,420] Deleting segment 25178 (kafka.log.Log)
[2018-10-18 12:57:19,187] Deleting segment 28754 (kafka.log.Log)
[2018-10-18 12:57:22,864] Deleting segment 32347 (kafka.log.Log)
^ Now the time difference has increased to ~1 hour 50 minutes (segment with
offset 25178)


[2018-10-18 13:15:24,665] Rolled new log segment at offset 39512 in 3 ms.
(kafka.log.Log)
[2018-10-18 13:47:39,469] Rolled new log segment at offset 43067 in 1 ms.
(kafka.log.Log)
[2018-10-18 14:20:38,673] Rolled new log segment at offset 46654 in 1 ms.
(kafka.log.Log)
[2018-10-18 14:53:07,311] Rolled new log segment at offset 50244 in 1 ms.
(kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 0 (kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 21583 (kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 25178 (kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 28754 (kafka.log.Log)
[2018-10-18 14:56:35,313] Deleting segment 32347 (kafka.log.Log)
[2018-10-18 14:56:40,024] Deleting segment 35913 (kafka.log.Log)
[2018-10-18 14:56:43,125] Deleting segment 39512 (kafka.log.Log)
[2018-10-18 14:56:44,951] Deleting segment 43067 (kafka.log.Log)
[2018-10-18 14:56:47,398] Deleting segment 46654 (kafka.log.Log)
^ Time difference between roll over and deletion ~1 hour 40 minutes
(segment with offset 25178)


[2018-10-18 15:25:51,075] Rolled new log segment at offset 53832 in 15 ms.
(kafka.log.Log)
[2018-10-18 15:58:02,546] Rolled new log segment at offset 57386 in 46 ms.
(kafka.log.Log)
[2018-10-18 16:30:16,005] Rolled new log segment at offset 60955 in 1 ms.
(kafka.log.Log)
[2018-10-18 17:02:48,638] Rolled new log segment at offset 64507 in 1 ms.
(kafka.log.Log)
[2018-10-18 17:35:47,851] Rolled new log segment at offset 68106 in 1 ms.
(kafka.log.Log)
[2018-10-18 18:08:25,337] Rolled new log segment at offset 71693 in 2 ms.
(kafka.log.Log)
^ No deletion even after 2 hour 30 minutes.

Any idea what is going on.?
Also note that these are for internal topics, and the properties configured
are

props.put(StreamsConfig.topicPrefix(TopicConfig.SEGMENT_BYTES_CONFIG),
1073741824);
props.put(StreamsConfig.topicPrefix(TopicConfig.RETENTION_BYTES_CONFIG),
1073741824);
props.put(StreamsConfig.topicPrefix(TopicConfig.RETENTION_MS_CONFIG), 20 *
60 * 1000);
props.put(StreamsConfig.topicPrefix(TopicConfig.DELETE_RETENTION_MS_CONFIG),
1000);

I plan to add
props.put(StreamsConfig.topicPrefix(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG),
"LogAppendTime");
since the message doesn't have CreationTime in it.
But I still wonder why the delay between roll over and deletion keeps
increasing and why the earlier deleted segments are being deleted again (at
least in the logs)

Thanks,
Vishnu

Re: Strange log delete behavior after segment rollover

Posted by Vishnu Viswanath <vi...@gmail.com>.
Sorry, Please ignore the question of deleting older segments again. Those
were for other partitions.
The only question is about the delay.

Thanks,
Vishnu

On Thu, Oct 18, 2018 at 1:26 PM Vishnu Viswanath <
vishnu.viswanath25@gmail.com> wrote:

> Hi All,
>
> I am seeing some strange behavior on how the rolled over segments are
> deleted. Here is the log for one of the partition.
>
> [2018-10-18 07:49:31,557] Rolled new log segment at offset 3600 in 24 ms.
> (kafka.log.Log)
> [2018-10-18 07:51:32,443] Deleting segment 0 (kafka.log.Log)
> ^ Log was rolled over and deleted within 2 minutes. Which is good.
>
> [2018-10-18 08:22:32,131] Rolled new log segment at offset 7216 in 261 ms.
> (kafka.log.Log)
> [2018-10-18 08:31:44,928] Deleting segment 0 (kafka.log.Log)
> [2018-10-18 08:31:44,928] Deleting segment 3600 (kafka.log.Log)
> ^ Log was rolled over and deleted within 2 minutes. But, why kafka trying
> to delete segment 0 again?
>
>
> [2018-10-18 08:55:10,442] Rolled new log segment at offset 10816 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 09:27:45,567] Rolled new log segment at offset 14420 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 09:37:07,728] Deleting segment 0 (kafka.log.Log)
> [2018-10-18 09:37:11,549] Deleting segment 7216 (kafka.log.Log)
> [2018-10-18 09:37:14,356] Deleting segment 10816 (kafka.log.Log)
> ^ Again same behavior, but the time between rollover and deletion has
> increased to 30 minutes.
>
>
> [2018-10-18 10:00:37,208] Rolled new log segment at offset 18016 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 10:32:43,677] Rolled new log segment at offset 21583 in 6 ms.
> (kafka.log.Log)
> [2018-10-18 10:35:28,808] Deleting segment 0 (kafka.log.Log)
> [2018-10-18 10:35:28,808] Deleting segment 7216 (kafka.log.Log)
> [2018-10-18 10:35:28,809] Deleting segment 10816 (kafka.log.Log)
> [2018-10-18 10:35:32,035] Deleting segment 14420 (kafka.log.Log)
> [2018-10-18 10:35:33,953] Deleting segment 18016 (kafka.log.Log)
> ^ Again deletion is happening after about 35 mins.
>
>
> [2018-10-18 11:05:18,246] Rolled new log segment at offset 25178 in 12 ms.
> (kafka.log.Log)
> [2018-10-18 11:37:55,385] Rolled new log segment at offset 28754 in 8 ms.
> (kafka.log.Log)
> [2018-10-18 12:10:20,695] Rolled new log segment at offset 32347 in 25 ms.
> (kafka.log.Log)
> [2018-10-18 12:43:07,535] Rolled new log segment at offset 35913 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 12:57:10,200] Deleting segment 0 (kafka.log.Log)
> [2018-10-18 12:57:10,200] Deleting segment 14420 (kafka.log.Log)
> [2018-10-18 12:57:10,201] Deleting segment 18016 (kafka.log.Log)
> [2018-10-18 12:57:12,584] Deleting segment 21583 (kafka.log.Log)
> [2018-10-18 12:57:16,420] Deleting segment 25178 (kafka.log.Log)
> [2018-10-18 12:57:19,187] Deleting segment 28754 (kafka.log.Log)
> [2018-10-18 12:57:22,864] Deleting segment 32347 (kafka.log.Log)
> ^ Now the time difference has increased to ~1 hour 50 minutes (segment
> with offset 25178)
>
>
> [2018-10-18 13:15:24,665] Rolled new log segment at offset 39512 in 3 ms.
> (kafka.log.Log)
> [2018-10-18 13:47:39,469] Rolled new log segment at offset 43067 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 14:20:38,673] Rolled new log segment at offset 46654 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 14:53:07,311] Rolled new log segment at offset 50244 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 14:56:35,313] Deleting segment 0 (kafka.log.Log)
> [2018-10-18 14:56:35,313] Deleting segment 21583 (kafka.log.Log)
> [2018-10-18 14:56:35,313] Deleting segment 25178 (kafka.log.Log)
> [2018-10-18 14:56:35,313] Deleting segment 28754 (kafka.log.Log)
> [2018-10-18 14:56:35,313] Deleting segment 32347 (kafka.log.Log)
> [2018-10-18 14:56:40,024] Deleting segment 35913 (kafka.log.Log)
> [2018-10-18 14:56:43,125] Deleting segment 39512 (kafka.log.Log)
> [2018-10-18 14:56:44,951] Deleting segment 43067 (kafka.log.Log)
> [2018-10-18 14:56:47,398] Deleting segment 46654 (kafka.log.Log)
> ^ Time difference between roll over and deletion ~1 hour 40 minutes
> (segment with offset 25178)
>
>
> [2018-10-18 15:25:51,075] Rolled new log segment at offset 53832 in 15 ms.
> (kafka.log.Log)
> [2018-10-18 15:58:02,546] Rolled new log segment at offset 57386 in 46 ms.
> (kafka.log.Log)
> [2018-10-18 16:30:16,005] Rolled new log segment at offset 60955 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 17:02:48,638] Rolled new log segment at offset 64507 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 17:35:47,851] Rolled new log segment at offset 68106 in 1 ms.
> (kafka.log.Log)
> [2018-10-18 18:08:25,337] Rolled new log segment at offset 71693 in 2 ms.
> (kafka.log.Log)
> ^ No deletion even after 2 hour 30 minutes.
>
> Any idea what is going on.?
> Also note that these are for internal topics, and the properties
> configured are
>
> props.put(StreamsConfig.topicPrefix(TopicConfig.SEGMENT_BYTES_CONFIG),
> 1073741824);
> props.put(StreamsConfig.topicPrefix(TopicConfig.RETENTION_BYTES_CONFIG),
> 1073741824);
> props.put(StreamsConfig.topicPrefix(TopicConfig.RETENTION_MS_CONFIG), 20 *
> 60 * 1000);
> props.put(StreamsConfig.topicPrefix(TopicConfig.DELETE_RETENTION_MS_CONFIG),
> 1000);
>
> I plan to add
> props.put(StreamsConfig.topicPrefix(TopicConfig.MESSAGE_TIMESTAMP_TYPE_CONFIG),
> "LogAppendTime");
> since the message doesn't have CreationTime in it.
> But I still wonder why the delay between roll over and deletion keeps
> increasing and why the earlier deleted segments are being deleted again (at
> least in the logs)
>
> Thanks,
> Vishnu
>