You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Manikumar (JIRA)" <ji...@apache.org> on 2017/09/07 19:28:00 UTC
[jira] [Resolved] (KAFKA-2623) Kakfa broker not deleting logs after
configured retention time properly
[ https://issues.apache.org/jira/browse/KAFKA-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Manikumar resolved KAFKA-2623.
------------------------------
Resolution: Fixed
Time-based log retention is enforced in KIP-33. Pl reopen if you think the issue still exists
> Kakfa broker not deleting logs after configured retention time properly
> -----------------------------------------------------------------------
>
> Key: KAFKA-2623
> URL: https://issues.apache.org/jira/browse/KAFKA-2623
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 0.8.2.0
> Environment: DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=12.04
> DISTRIB_CODENAME=precise
> DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
> NAME="Ubuntu"
> VERSION="12.04.5 LTS, Precise Pangolin"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu precise (12.04.5 LTS)"
> VERSION_ID="12.04"
> Reporter: Hao Zhang
> Assignee: Jay Kreps
>
> Context:
> To get an accurate estimate on how much retention we have for each topic/partition, we have a cron job iterating each topic/partition folder on each node of a cluster, measuring the timestamp difference between the newest and oldest log files.
> Problem:
> We notice that it's very common that between leaders and followers, the time differences are vastly different. On the leader the timestamp differences are normally about a week (our retention policy), but on the follower the timestamp differences can sometimes range between just a few hours to 2-3 days.
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001536840178.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001537497855.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538155208.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001538811692.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001539468154.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540122891.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001540775681.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001541430669.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542088333.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:48 00000000001542746722.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001543405006.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544062197.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001544718413.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001545374173.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546029145.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001546686144.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001547344190.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548001698.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001548657672.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549312958.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:49 00000000001549969014.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001550623380.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551279821.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001551937920.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001552597354.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553256336.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001553914505.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001554571426.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555228277.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001555882081.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001556538902.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:50 00000000001557196332.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001557852974.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001558510709.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559166839.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001559823667.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001560478631.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561136505.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001561792222.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001562450149.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563107321.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:51 00000000001563763826.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001564420526.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565076456.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001565735877.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001566394151.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567051743.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001567709678.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001568366114.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 05:52 00000000001569022963.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 08:09 00000000001569681970.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 10:37 00000000001570340180.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 13:05 00000000001570997358.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 4 16:11 00000000001571654289.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 01:56 00000000001572310787.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 05:20 00000000001572967484.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 08:19 00000000001573626503.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 11:02 00000000001574284909.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 13:49 00000000001574944379.log
> -rw-rw-r-- 1 kloak kloak 256M Oct 5 18:26 00000000001575603057.log
> -rw-rw-r-- 1 kloak kloak 28M Oct 5 21:02 00000000001576258146.log
> Observation:
> We noticed that there are a lot of index files with size 200. The modified timestamp of these index files are pretty much identical.
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001536840178.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001537497855.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538155208.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001538811692.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001539468154.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540122891.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001540775681.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001541430669.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542088333.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:48 00000000001542746722.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001543405006.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544062197.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001544718413.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001545374173.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546029145.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001546686144.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001547344190.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548001698.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001548657672.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549312958.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:49 00000000001549969014.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001550623380.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551279821.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001551937920.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001552597354.index
> -rw-rw-r-- 1 kloak kloak 200 Oct 4 05:50 00000000001553256336.index
> Theory:
> Our guess is that the broker is trying to delete these log and index files, thus the timestamps are all modified at pretty much the same time. However, for some reason the deletion doesn't succeed and thus leave all the log/index files with the same timestamp. But there is no log deletion error in the log whatsoever.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)