1 topic that keeps growing indefinitely

[Wim Van Leuven <wi...@highestpoint.biz>]

Hello,

we have a 3 node kafka cluster setup with quite a bunch of topics that have
a nice life: regular cleans, compact, etc. 1 topic however keeps growing
indefinitely. New segments are recreated at regular intervals, but old
segments are never deleted.

Our setup is based on Confluent 3.2.0 OSS, hence Apache Kafka 0.10.2.0. The
server.props related to log retention are fairly straightforward:

log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.retention.check.interval.ms=300000
log.retention.bytes=-1
log.retention.ms=7200000
log.segment.bytes=1073741824

We do apply specific configurations for each topic to ensure proper
lifetime, cleanup etc. For the topic at hand, those are

retention.ms=3600001
segment.ms=3600000
compression.type=uncompressed
cleanup.policy=delete

I tried quite a bunch of configurations to try various setups, but I'm in a
dead end and my final resort would be to delete the topic (and all data
underneath) and just recreate. I still can because this is a test/staging
environment. I'm a bit reluctant as I'd like to understand what's going on
because if this hits us on a production system ...

Any help or insights would be much appreciated!
-wim