You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "huxihx (Jira)" <ji...@apache.org> on 2020/07/24 01:38:00 UTC
[jira] [Resolved] (KAFKA-10268) dynamic config like
"--delete-config log.retention.ms" doesn't work
[ https://issues.apache.org/jira/browse/KAFKA-10268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
huxihx resolved KAFKA-10268.
----------------------------
Fix Version/s: 2.7.0
Resolution: Fixed
> dynamic config like "--delete-config log.retention.ms" doesn't work
> -------------------------------------------------------------------
>
> Key: KAFKA-10268
> URL: https://issues.apache.org/jira/browse/KAFKA-10268
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner
> Affects Versions: 2.1.1
> Reporter: zhifeng.peng
> Assignee: huxihx
> Priority: Major
> Fix For: 2.7.0
>
> Attachments: server.log.2020-07-13-14
>
>
> After I set "log.retention.ms=301000" to clean the data,i use the cmd
> "bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type brokers --entity-default --alter --delete-config log.retention.ms" to reset to default.
> Static broker configuration like log.retention.hours is 168h and no topic level configuration like retention.ms.
> it did not take effect actually although server.log print the broker configuration like that.
> log.retention.check.interval.ms = 300000
> log.retention.hours = 168
> log.retention.minutes = null
> {color:#ff0000}log.retention.ms = null{color}
> log.roll.hours = 168
> log.roll.jitter.hours = 0
> log.roll.jitter.ms = null
> log.roll.ms = null
> log.segment.bytes = 1073741824
> log.segment.delete.delay.ms = 60000
>
> Then we can see that retention time is still 301000ms from the server.log and segments have been deleted.
> [2020-07-13 14:30:00,958] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Found deletable segments with base offsets [5005329,6040360] due to retention time 301000ms breach (kafka.log.Log)
> [2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 5005329, size 1073741222] for deletion. (kafka.log.Log)
> [2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 6040360, size 1073728116] for deletion. (kafka.log.Log)
> [2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Incrementing log start offset to 7075648 (kafka.log.Log)
> [2020-07-13 14:30:00,960] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Found deletable segments with base offsets [5005330,6040410] {color:#FF0000}due to retention time 301000ms{color} breach (kafka.log.Log)
> [2020-07-13 14:30:00,960] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 5005330, size 1073732368] for deletion. (kafka.log.Log)
> [2020-07-13 14:30:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 6040410, size 1073735366] for deletion. (kafka.log.Log)
> [2020-07-13 14:30:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Incrementing log start offset to 7075685 (kafka.log.Log)
> [2020-07-13 14:31:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Deleting segment 5005329 (kafka.log.Log)
> [2020-07-13 14:31:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Deleting segment 6040360 (kafka.log.Log)
> [2020-07-13 14:31:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Deleting segment 5005330 (kafka.log.Log)
> [2020-07-13 14:31:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Deleting segment 6040410 (kafka.log.Log)
> [2020-07-13 14:31:01,144] INFO Deleted log /data/kafka_logs-test/test_retention-2/00000000000006040360.log.deleted. (kafka.log.LogSegment)
> [2020-07-13 14:31:01,144] INFO Deleted offset index /data/kafka_logs-test/test_retention-2/00000000000006040360.index.deleted. (kafka.log.LogSegment)
> [2020-07-13 14:31:01,144] INFO Deleted time index /data/kafka_logs-test/test_retention-2/00000000000006040360.timeindex.deleted. (kafka.log.LogSegment)
>
> Here are a few steps to reproduce it.
> 1、set log.retention.ms=301000:
> bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type brokers --entity-default --alter --add-config log.retention.ms=301000
> 2、produce messages to the topic:
> bin/kafka-producer-perf-test.sh --topic test_retention --num-records 10000000 --throughput -1 --producer-props bootstrap.servers=10.129.104.15:9092 --record-size 1024
> 3、reset log.retention.ms to the default:
> bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type brokers --entity-default --alter --delete-config log.retention.ms
>
> I have attched server.log. You can see the log from row 238 to row 731.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)