Automating topic cleanup

[Diogo Vieira <di...@kelvininc.com>]

Hello!

I’m using a Kafka cluster with dynamically created and temporary topics that each map to a different consumer. After a consumer disconnects the topic isn’t cleaned which means my topics will grow with the consumers.

As far as I know, Kafka won’t automatically clean those topics (I believe something like it was suggested as a feature when topic deletion was introduced but was never implemented).

Does anyone more knowledgeable than me can please provide some insight to tackle this problem? I’m thinking of automating the cleanup of these topics, but I have no idea how to find out which topics should be deleted, since I would like to have access to the timestamp of the last consumer offset commit (or something like it) to clean topics after an arbitrary amount of time.

Thank you in advance,
Diogo Vieira