You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Hao Zhang (JIRA)" <ji...@apache.org> on 2015/10/08 01:10:26 UTC

[jira] [Created] (KAFKA-2623) Kakfa broker not deleting logs after configured retention time properly

Hao Zhang created KAFKA-2623:
--------------------------------

             Summary: Kakfa broker not deleting logs after configured retention time properly
                 Key: KAFKA-2623
                 URL: https://issues.apache.org/jira/browse/KAFKA-2623
             Project: Kafka
          Issue Type: Bug
          Components: log
    Affects Versions: 0.8.2.0
         Environment: hzhang@kloak65-sjc1:/var/kafka-spool/data/dispatch.events.client_eyeballed-7$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
NAME="Ubuntu"
VERSION="12.04.5 LTS, Precise Pangolin"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu precise (12.04.5 LTS)"
VERSION_ID="12.04"
            Reporter: Hao Zhang
            Assignee: Jay Kreps


Context:
To get an accurate estimate on how much retention we have for each topic/partition, we have a cron job iterating each topic/partition folder on each node of a cluster, measuring the timestamp difference between the newest and oldest log files. 

Problem:
We notice that it's very common that between leaders and followers, the time differences are vastly different. On the leader the timestamp differences are normally about a week (our retention policy), but on the follower the timestamp differences can sometimes range between just a few hours to 2-3 days.

hzhang@kloak65-sjc1:/var/kafka-spool/data/dispatch.events.client_eyeballed-7$ ls -lrht *.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001536840178.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001537497855.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001538155208.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001538811692.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001539468154.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001540122891.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001540775681.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001541430669.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001542088333.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:48 00000000001542746722.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001543405006.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001544062197.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001544718413.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001545374173.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001546029145.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001546686144.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001547344190.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001548001698.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001548657672.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001549312958.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:49 00000000001549969014.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001550623380.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001551279821.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001551937920.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001552597354.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001553256336.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001553914505.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001554571426.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001555228277.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001555882081.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001556538902.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:50 00000000001557196332.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001557852974.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001558510709.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001559166839.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001559823667.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001560478631.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001561136505.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001561792222.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001562450149.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001563107321.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:51 00000000001563763826.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 00000000001564420526.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 00000000001565076456.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 00000000001565735877.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 00000000001566394151.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 00000000001567051743.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 00000000001567709678.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 00000000001568366114.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 05:52 00000000001569022963.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 08:09 00000000001569681970.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 10:37 00000000001570340180.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 13:05 00000000001570997358.log
-rw-rw-r-- 1 kloak kloak 256M Oct  4 16:11 00000000001571654289.log
-rw-rw-r-- 1 kloak kloak 256M Oct  5 01:56 00000000001572310787.log
-rw-rw-r-- 1 kloak kloak 256M Oct  5 05:20 00000000001572967484.log
-rw-rw-r-- 1 kloak kloak 256M Oct  5 08:19 00000000001573626503.log
-rw-rw-r-- 1 kloak kloak 256M Oct  5 11:02 00000000001574284909.log
-rw-rw-r-- 1 kloak kloak 256M Oct  5 13:49 00000000001574944379.log
-rw-rw-r-- 1 kloak kloak 256M Oct  5 18:26 00000000001575603057.log
-rw-rw-r-- 1 kloak kloak  28M Oct  5 21:02 00000000001576258146.log

Observation:
We noticed that there are a lot of index files with size 200. The modified timestamp of these index files are pretty much identical.

hzhang@kloak65-sjc1:/var/kafka-spool/data/dispatch.events.client_eyeballed-7$ ls -lhrt *.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001536840178.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001537497855.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001538155208.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001538811692.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001539468154.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001540122891.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001540775681.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001541430669.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001542088333.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:48 00000000001542746722.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001543405006.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001544062197.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001544718413.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001545374173.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001546029145.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001546686144.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001547344190.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001548001698.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001548657672.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001549312958.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:49 00000000001549969014.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:50 00000000001550623380.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:50 00000000001551279821.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:50 00000000001551937920.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:50 00000000001552597354.index
-rw-rw-r-- 1 kloak kloak  200 Oct  4 05:50 00000000001553256336.index

Theory:
Our guess is that the broker is trying to delete these log and index files, thus the timestamps are all modified at pretty much the same time. However, for some reason the deletion doesn't succeed and thus leave all the log/index files with the same timestamp. But there is no log deletion error in the log whatsoever.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)