You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Asko Alhoniemi <as...@lekane.com> on 2017/10/06 13:17:17 UTC
Compacted topic, entry deletion, kafka 0.11.0.0
Hello
I understand that the compacted topic is meant to keep at least the latest
key value pair.
However, I am having an issue since it can happen that entry becomes old
and I need to remove it. It may also occur, that I am not able to send key
"null" pair. So I need another method to remove my hanging entries. My hope
was with the following configuration:
Topic:test.topic PartitionCount:30 ReplicationFactor:2
Configs:segment.bytes=1048576,min.cleanable.dirty.ratio=0.1,
delete.retention.ms=1800000,retention.ms=900000,segment.ms
=900000,cleanup.policy=compact,delete
Topic: test.topic Partition: 0 Leader: 0 Replicas: 1,0 Isr:
0
Topic: test.topic Partition: 1 Leader: 0 Replicas: 0,1 Isr:
0
Topic: test.topic Partition: 2 Leader: 0 Replicas: 1,0 Isr:
0
Topic: test.topic Partition: 3 Leader: 0 Replicas: 0,1 Isr:
0
Topic: test.topic Partition: 4 Leader: 0 Replicas: 1,0 Isr:
0
Topic: test.topic Partition: 5 Leader: 0 Replicas: 0,1 Isr:
0
Topic: test.topic Partition: 6 Leader: 0 Replicas: 1,0 Isr:
0
Topic: test.topic Partition: 7 Leader: 0 Replicas: 0,1 Isr:
0
Topic: test.topic Partition: 8 Leader: 0 Replicas: 1,0 Isr:
0
Topic: test.topic Partition: 9 Leader: 0 Replicas: 0,1 Isr:
0
Topic: test.topic Partition: 10 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 11 Leader: 0 Replicas: 0,1
Isr: 0
Topic: test.topic Partition: 12 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 13 Leader: 0 Replicas: 0,1
Isr: 0
Topic: test.topic Partition: 14 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 15 Leader: 0 Replicas: 0,1
Isr: 0
Topic: test.topic Partition: 16 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 17 Leader: 0 Replicas: 0,1
Isr: 0
Topic: test.topic Partition: 18 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 19 Leader: 0 Replicas: 0,1
Isr: 0
Topic: test.topic Partition: 20 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 21 Leader: 0 Replicas: 0,1
Isr: 0
Topic: test.topic Partition: 22 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 23 Leader: 0 Replicas: 0,1
Isr: 0
Topic: test.topic Partition: 24 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 25 Leader: 0 Replicas: 0,1
Isr: 0
Topic: test.topic Partition: 26 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 27 Leader: 0 Replicas: 0,1
Isr: 0
Topic: test.topic Partition: 28 Leader: 0 Replicas: 1,0
Isr: 0
Topic: test.topic Partition: 29 Leader: 0 Replicas: 0,1
Isr: 0
but it does not get cleaned. I stop writing one key value pair, but still
can see that key value pair after days.
Then I changed broker configuration on the fly:
# The minimum age of a log file to be eligible for deletion
#log.retention.hours=168
log.retention.minutes=16
# A size-based retention policy for logs. Segments are pruned from the log
as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824
log.retention.bytes=10000
# The maximum size of a log segment file. When this size is reached a new
log segment will be created.
#log.segment.bytes=1073741824
log.segment.bytes=10000
So far I haven't been able to drop away the desired key value pair with any
of these configuration actions.
Am I trying something that is not possible or am I missing some
configuration options or should this work?
br.
Asko Alhoniemi
Re: Compacted topic, entry deletion, kafka 0.11.0.0
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Setting topic policy to "compact,delete" should be sufficient. Cf.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-71%3A+Enable+log+compaction+and+deletion+to+co-exist
Note: retention time is not based on wall-clock time, but embedded
record timestamps. Thus, old messages get only deleted if new messages
with larger timestamps are written to the topic.
-Matthias
On 10/6/17 6:17 AM, Asko Alhoniemi wrote:
> Hello
>
> I understand that the compacted topic is meant to keep at least the latest
> key value pair.
>
> However, I am having an issue since it can happen that entry becomes old
> and I need to remove it. It may also occur, that I am not able to send key
> "null" pair. So I need another method to remove my hanging entries. My hope
> was with the following configuration:
>
> Topic:test.topic PartitionCount:30 ReplicationFactor:2
> Configs:segment.bytes=1048576,min.cleanable.dirty.ratio=0.1,
> delete.retention.ms=1800000,retention.ms=900000,segment.ms
> =900000,cleanup.policy=compact,delete
> Topic: test.topic Partition: 0 Leader: 0 Replicas: 1,0 Isr:
> 0
> Topic: test.topic Partition: 1 Leader: 0 Replicas: 0,1 Isr:
> 0
> Topic: test.topic Partition: 2 Leader: 0 Replicas: 1,0 Isr:
> 0
> Topic: test.topic Partition: 3 Leader: 0 Replicas: 0,1 Isr:
> 0
> Topic: test.topic Partition: 4 Leader: 0 Replicas: 1,0 Isr:
> 0
> Topic: test.topic Partition: 5 Leader: 0 Replicas: 0,1 Isr:
> 0
> Topic: test.topic Partition: 6 Leader: 0 Replicas: 1,0 Isr:
> 0
> Topic: test.topic Partition: 7 Leader: 0 Replicas: 0,1 Isr:
> 0
> Topic: test.topic Partition: 8 Leader: 0 Replicas: 1,0 Isr:
> 0
> Topic: test.topic Partition: 9 Leader: 0 Replicas: 0,1 Isr:
> 0
> Topic: test.topic Partition: 10 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 11 Leader: 0 Replicas: 0,1
> Isr: 0
> Topic: test.topic Partition: 12 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 13 Leader: 0 Replicas: 0,1
> Isr: 0
> Topic: test.topic Partition: 14 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 15 Leader: 0 Replicas: 0,1
> Isr: 0
> Topic: test.topic Partition: 16 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 17 Leader: 0 Replicas: 0,1
> Isr: 0
> Topic: test.topic Partition: 18 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 19 Leader: 0 Replicas: 0,1
> Isr: 0
> Topic: test.topic Partition: 20 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 21 Leader: 0 Replicas: 0,1
> Isr: 0
> Topic: test.topic Partition: 22 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 23 Leader: 0 Replicas: 0,1
> Isr: 0
> Topic: test.topic Partition: 24 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 25 Leader: 0 Replicas: 0,1
> Isr: 0
> Topic: test.topic Partition: 26 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 27 Leader: 0 Replicas: 0,1
> Isr: 0
> Topic: test.topic Partition: 28 Leader: 0 Replicas: 1,0
> Isr: 0
> Topic: test.topic Partition: 29 Leader: 0 Replicas: 0,1
> Isr: 0
>
> but it does not get cleaned. I stop writing one key value pair, but still
> can see that key value pair after days.
>
> Then I changed broker configuration on the fly:
>
> # The minimum age of a log file to be eligible for deletion
> #log.retention.hours=168
> log.retention.minutes=16
>
> # A size-based retention policy for logs. Segments are pruned from the log
> as long as the remaining
> # segments don't drop below log.retention.bytes.
> #log.retention.bytes=1073741824
> log.retention.bytes=10000
>
> # The maximum size of a log segment file. When this size is reached a new
> log segment will be created.
> #log.segment.bytes=1073741824
> log.segment.bytes=10000
>
> So far I haven't been able to drop away the desired key value pair with any
> of these configuration actions.
>
> Am I trying something that is not possible or am I missing some
> configuration options or should this work?
>
> br.
>
> Asko Alhoniemi
>