You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "zhifeng.peng (Jira)" <ji...@apache.org> on 2020/07/13 11:02:00 UTC
[jira] [Created] (KAFKA-10268) dynamic config like "--delete-config
log.retention.ms" not work
zhifeng.peng created KAFKA-10268:
------------------------------------
Summary: dynamic config like "--delete-config log.retention.ms" not work
Key: KAFKA-10268
URL: https://issues.apache.org/jira/browse/KAFKA-10268
Project: Kafka
Issue Type: Bug
Components: log, log cleaner
Affects Versions: 2.1.1
Reporter: zhifeng.peng
Attachments: server.log.2020-07-13-14
After I set "log.retention.ms=301000" to clean the data,i use the cmd "bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type brokers --entity-default --alter --delete-config log.retention.ms" to reset to default. Static broker configuration like log.retention.hours is 168h and no topic level configuration like retention.ms.
But it did not take effect actually although server.log print the broker configuration like that.
log.retention.check.interval.ms = 300000
log.retention.hours = 168
log.retention.minutes = null
{color:#FF0000}log.retention.ms = null{color}
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 60000
Then we can see that retention time is still 301000ms from the server.log and segments have been deleted.
[2020-07-13 14:30:00,958] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Found deletable segments with base offsets [5005329,6040360] due to retention time 301000ms breach (kafka.log.Log)
[2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 5005329, size 1073741222] for deletion. (kafka.log.Log)
[2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 6040360, size 1073728116] for deletion. (kafka.log.Log)
[2020-07-13 14:30:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Incrementing log start offset to 7075648 (kafka.log.Log)
[2020-07-13 14:30:00,960] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Found deletable segments with base offsets [5005330,6040410] due to retention time 301000ms breach (kafka.log.Log)
[2020-07-13 14:30:00,960] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 5005330, size 1073732368] for deletion. (kafka.log.Log)
[2020-07-13 14:30:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Scheduling log segment [baseOffset 6040410, size 1073735366] for deletion. (kafka.log.Log)
[2020-07-13 14:30:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Incrementing log start offset to 7075685 (kafka.log.Log)
[2020-07-13 14:31:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Deleting segment 5005329 (kafka.log.Log)
[2020-07-13 14:31:00,959] INFO [Log partition=test_retention-2, dir=/data/kafka_logs-test] Deleting segment 6040360 (kafka.log.Log)
[2020-07-13 14:31:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Deleting segment 5005330 (kafka.log.Log)
[2020-07-13 14:31:00,961] INFO [Log partition=test_retention-0, dir=/data/kafka_logs-test] Deleting segment 6040410 (kafka.log.Log)
[2020-07-13 14:31:01,144] INFO Deleted log /data/kafka_logs-test/test_retention-2/00000000000006040360.log.deleted. (kafka.log.LogSegment)
[2020-07-13 14:31:01,144] INFO Deleted offset index /data/kafka_logs-test/test_retention-2/00000000000006040360.index.deleted. (kafka.log.LogSegment)
[2020-07-13 14:31:01,144] INFO Deleted time index /data/kafka_logs-test/test_retention-2/00000000000006040360.timeindex.deleted. (kafka.log.LogSegment)
Here are a few steps to reproduce it.
1、set log.retention.ms=301000:
bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type brokers --entity-default --alter --add-config log.retention.ms=301000
2、produce messages to the topic:
bin/kafka-producer-perf-test.sh --topic test_retention --num-records 10000000 --throughput -1 --producer-props bootstrap.servers=10.129.104.15:9092 --record-size 1024
3、reset log.retention.ms to the default:
bin/kafka-configs.sh --bootstrap-server 10.129.104.15:9092 --entity-type brokers --entity-default --alter --delete-config log.retention.ms
I have attched server.log. You can see the log from row 238 to row 731.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)