You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Linyue Li <Li...@microsoft.com.INVALID> on 2018/08/17 11:18:44 UTC
changelog Topic can't be deleted even if enable "compact,delete"
cleanup.policy for Kafka 1.0.0
Hi Team,
We found that our changelog topic can't be deleted even if we enable the both "compact,delete" policy on it.
The changelog topic is for the window store, since we know that the session store not support "compact,delete" policy now. (https://issues.apache.org/jira/browse/KAFKA-7101) But from the kafka side, we found that even the window store changelog hasn't been deleted and its size increase time by time.
The topic name is "AuditTrailRecord_PROD0-dedup-raw-store-changelog", and the configuration for this topic is list as below:
Topic config:
Cleanup.policy=compact,delete
Retention.ms=10800000
All the other configs are default values, and our kafka version is 1.0.0.
The retention time is only 3 hours, but we can still see the old segments even on 08/13, far exceed the retention.ms, and there are many old segments in one partition. And we found the kafka logs not contains "delete" info for the related topic.
We know kafka support the co-exist cleanup policy since 0.10.0, so it should work for 1.0.0. And all the other topics with only one cleanup.policy "delete" worked as expected. So could you help us to find what happens. Thanks!
Segments under topic partition folder:
/data/ssd0/kafka-logs/AuditTrailRecord_PROD0-dedup-raw-store-changelog-8$ ls -all
-rw-r--r-- 1 root root 756016 Aug 13 01:30 00000000000088449555.index
-rw-r--r-- 1 root root 1073727918 Aug 13 01:30 00000000000088449555.log
-rw-r--r-- 1 root root 777996 Aug 13 01:30 00000000000088449555.timeindex
-rw-r--r-- 1 root root 34 Aug 12 19:24 00000000000088449555.txnindex
-rw-r--r-- 1 root root 604672 Aug 13 06:16 00000000000089887963.index
-rw-r--r-- 1 root root 1073739359 Aug 13 06:16 00000000000089887963.log
-rw-r--r-- 1 root root 474660 Aug 13 06:16 00000000000089887963.timeindex
-rw-r--r-- 1 root root 584960 Aug 13 09:13 00000000000091333539.index
-rw-r--r-- 1 root root 1073739351 Aug 13 09:13 00000000000091333539.log
-rw-r--r-- 1 root root 379476 Aug 13 09:13 00000000000091333539.timeindex
-rw-r--r-- 1 root root 594728 Aug 13 13:08 00000000000092779354.index
-rw-r--r-- 1 root root 1073729489 Aug 13 13:08 00000000000092779354.log
-rw-r--r-- 1 root root 458172 Aug 13 13:08 00000000000092779354.timeindex
-rw-r--r-- 1 root root 34 Aug 13 11:00 00000000000092779354.txnindex
-rw-r--r-- 1 root root 627928 Aug 13 21:48 00000000000094224193.index
-rw-r--r-- 1 root root 1073729918 Aug 13 21:48 00000000000094224193.log
-rw-r--r-- 1 root root 599664 Aug 13 21:48 00000000000094224193.timeindex
-rw-r--r-- 1 root root 606608 Aug 14 03:55 00000000000095666017.index
-rw-r--r-- 1 root root 1073740608 Aug 14 03:55 00000000000095666017.log
-rw-r--r-- 1 root root 506148 Aug 14 03:55 00000000000095666017.timeindex
-rw-r--r-- 1 root root 590344 Aug 14 07:22 00000000000097112178.index
-rw-r--r-- 1 root root 1073737347 Aug 14 07:22 00000000000097112178.log
-rw-r--r-- 1 root root 433368 Aug 14 07:22 00000000000097112178.timeindex
-rw-r--r-- 1 root root 591624 Aug 14 10:32 00000000000098559295.index
-rw-r--r-- 1 root root 1073729690 Aug 14 10:32 00000000000098559295.log
-rw-r--r-- 1 root root 416652 Aug 14 10:32 00000000000098559295.timeindex
-rw-r--r-- 1 root root 597408 Aug 14 14:33 00000000000100005116.index
-rw-r--r-- 1 root root 1073733063 Aug 14 14:33 00000000000100005116.log
-rw-r--r-- 1 root root 447312 Aug 14 14:33 00000000000100005116.timeindex
-rw-r--r-- 1 root root 34 Aug 14 13:01 00000000000100005116.txnindex
-rw-r--r-- 1 root root 636000 Aug 15 01:00 00000000000101450083.index
-rw-r--r-- 1 root root 1073732865 Aug 15 01:00 00000000000101450083.log
-rw-r--r-- 1 root root 645204 Aug 15 01:00 00000000000101450083.timeindex
-rw-r--r-- 1 root root 608992 Aug 15 06:55 00000000000102891532.index
-rw-r--r-- 1 root root 1073740472 Aug 15 06:55 00000000000102891532.log
-rw-r--r-- 1 root root 519708 Aug 15 06:55 00000000000102891532.timeindex
-rw-r--r-- 1 root root 34 Aug 15 06:00 00000000000102891532.txnindex
-rw-r--r-- 1 root root 593344 Aug 15 10:24 00000000000104335942.index
-rw-r--r-- 1 root root 1073727177 Aug 15 10:24 00000000000104335942.log
-rw-r--r-- 1 root root 438120 Aug 15 10:24 00000000000104335942.timeindex
-rw-r--r-- 1 root root 597512 Aug 15 14:41 00000000000105781865.index
-rw-r--r-- 1 root root 1073733762 Aug 15 14:41 00000000000105781865.log
-rw-r--r-- 1 root root 444468 Aug 15 14:41 00000000000105781865.timeindex
-rw-r--r-- 1 root root 34 Aug 15 12:57 00000000000105781865.txnindex
-rw-r--r-- 1 root root 626128 Aug 16 00:09 00000000000107227432.index
-rw-r--r-- 1 root root 1073736231 Aug 16 00:09 00000000000107227432.log
-rw-r--r-- 1 root root 602748 Aug 16 00:09 00000000000107227432.timeindex
-rw-r--r-- 1 root root 605104 Aug 16 05:22 00000000000108669491.index
-rw-r--r-- 1 root root 1073741003 Aug 16 05:22 00000000000108669491.log
-rw-r--r-- 1 root root 501684 Aug 16 05:22 00000000000108669491.timeindex
-rw-r--r-- 1 root root 586408 Aug 16 08:17 00000000000110115744.index
-rw-r--r-- 1 root root 1073737543 Aug 16 08:17 00000000000110115744.log
-rw-r--r-- 1 root root 410532 Aug 16 08:17 00000000000110115744.timeindex
-rw-r--r-- 1 root root 591744 Aug 16 11:53 00000000000111563044.index
-rw-r--r-- 1 root root 1073731095 Aug 16 11:53 00000000000111563044.log
-rw-r--r-- 1 root root 432252 Aug 16 11:53 00000000000111563044.timeindex
-rw-r--r-- 1 root root 34 Aug 16 08:58 00000000000111563044.txnindex
-rw-r--r-- 1 root root 611144 Aug 16 18:04 00000000000113008214.index
-rw-r--r-- 1 root root 1073736959 Aug 16 18:04 00000000000113008214.log
-rw-r--r-- 1 root root 542856 Aug 16 18:04 00000000000113008214.timeindex
-rw-r--r-- 1 root root 619616 Aug 17 02:30 00000000000114452368.index
-rw-r--r-- 1 root root 1073729948 Aug 17 02:30 00000000000114452368.log
-rw-r--r-- 1 root root 56 Aug 16 18:04 00000000000114452368.snapshot
-rw-r--r-- 1 root root 573504 Aug 17 02:30 00000000000114452368.timeindex
-rw-r--r-- 1 root root 34 Aug 17 02:02 00000000000114452368.txnindex
-rw-r--r-- 1 root root 10485760 Aug 17 04:10 00000000000115896478.index
-rw-r--r-- 1 root root 291796329 Aug 17 04:10 00000000000115896478.log
-rw-r--r-- 1 root root 56 Aug 17 02:30 00000000000115896478.snapshot
-rw-r--r-- 1 root root 10485756 Aug 17 04:10 00000000000115896478.timeindex
-rw-r--r-- 1 root root 16 Aug 7 05:02 leader-epoch-checkpoint
Kafka logs for the changelog topic:
....
[2018-08-16 22:18:18,170] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-11' in 1 ms. (kafka.log.Log)
[2018-08-17 01:04:27,291] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-8' in 1 ms. (kafka.log.Log)
[2018-08-17 01:05:40,431] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-30' in 1 ms. (kafka.log.Log)
[2018-08-17 02:59:17,862] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-37' in 1 ms. (kafka.log.Log)
[2018-08-17 04:26:03,798] INFO [ReplicaFetcherManager on broker 3] Removed fetcher for partitions AuditTrailRecord_PROD0-dedup-raw-store-changelog-7 (kafka.server.ReplicaFetcherManager)
[2018-08-17 04:26:03,799] INFO [ReplicaFetcherManager on broker 3] Added fetcher for partitions List([AuditTrailRecord_PROD0-dedup-raw-store-changelog-7, initOffset 114887572 to broker BrokerEndPoint(9,25.68.1.18,9092)] ) (kafka.server.ReplicaFetcherManager)
[2018-08-17 04:26:03,900] INFO [ReplicaFetcher replicaId=3, leaderId=9, fetcherId=0] Based on follower's leader epoch, leader replied with an offset 114887572 >= the follower's log end offset 114887572 in AuditTrailRecord_PROD0-dedup-raw-store-changelog-7. No truncation needed. (kafka.server.ReplicaFetcherThread)
[2018-08-17 04:26:03,900] INFO Truncating AuditTrailRecord_PROD0-dedup-raw-store-changelog-7 to 114887572 has no effect as the largest offset in the log is 114887571. (kafka.log.Log)
[2018-08-17 04:28:40,322] INFO Updated PartitionLeaderEpoch. New: {epoch:31, offset:114887572}, Current: {epoch:30, offset99723767} for Partition: AuditTrailRecord_PROD0-dedup-raw-store-changelog-7. Cache now contains 1 entries. (kafka.server.epoch.LeaderEpochFileCache)
[2018-08-17 04:31:32,179] INFO [ReplicaFetcherManager on broker 3] Removed fetcher for partitions AuditTrailRecord_PROD0-dedup-raw-store-changelog-7 (kafka.server.ReplicaFetcherManager)
[2018-08-17 04:31:32,179] INFO [ReplicaFetcherManager on broker 3] Added fetcher for partitions List([AuditTrailRecord_PROD0-dedup-raw-store-changelog-7, initOffset 114887573 to broker BrokerEndPoint(1,25.72.80.146,9092)] ) (kafka.server.ReplicaFetcherManager)
[2018-08-17 04:31:32,201] INFO [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Based on follower's leader epoch, leader replied with an offset 114887573 >= the follower's log end offset 114887573 in AuditTrailRecord_PROD0-dedup-raw-store-changelog-7. No truncation needed. (kafka.server.ReplicaFetcherThread)
[2018-08-17 04:31:32,201] INFO Truncating AuditTrailRecord_PROD0-dedup-raw-store-changelog-7 to 114887573 has no effect as the largest offset in the log is 114887572. (kafka.log.Log)
[2018-08-17 04:31:32,253] INFO Incrementing log start offset of partition AuditTrailRecord_PROD0-dedup-raw-store-changelog-7 to 114171587 in dir /kafka3 (kafka.log.Log)
[2018-08-17 04:31:32,259] INFO Cleared earliest 0 entries from epoch cache based on passed offset 114171587 leaving 2 in EpochFile for partition AuditTrailRecord_PROD0-dedup-raw-store-changelog-7 (kafka.server.epoch.LeaderEpochFileCache)
[2018-08-17 04:38:19,615] INFO Updated PartitionLeaderEpoch. New: {epoch:32, offset:114887573}, Current: {epoch:31, offset114887572} for Partition: AuditTrailRecord_PROD0-dedup-raw-store-changelog-7. Cache now contains 2 entries. (kafka.server.epoch.LeaderEpochFileCache)
[2018-08-17 05:50:30,544] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-29' in 1 ms. (kafka.log.Log)
[2018-08-17 06:30:58,255] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-7' in 1 ms. (kafka.log.Log)
[2018-08-17 08:01:50,403] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-11' in 1 ms. (kafka.log.Log)
[2018-08-17 08:59:49,316] INFO Updated PartitionLeaderEpoch. New: {epoch:33, offset:115789932}, Current: {epoch:32, offset114887573} for Partition: AuditTrailRecord_PROD0-dedup-raw-store-changelog-7. Cache now contains 3 entries. (kafka.server.epoch.LeaderEpochFileCache)
[2018-08-17 08:59:54,004] INFO Updated PartitionLeaderEpoch. New: {epoch:34, offset:115790053}, Current: {epoch:33, offset115789932} for Partition: AuditTrailRecord_PROD0-dedup-raw-store-changelog-7. Cache now contains 4 entries. (kafka.server.epoch.LeaderEpochFileCache)
[2018-08-17 09:30:36,359] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-8' in 1 ms. (kafka.log.Log)
[2018-08-17 09:32:22,202] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-30' in 1 ms. (kafka.log.Log)
[2018-08-17 10:30:24,228] INFO Rolled new log segment for 'AuditTrailRecord_PROD0-dedup-raw-store-changelog-37' in 1 ms. (kafka.log.Log)