How to delete defunct topics

[Jagbir Hooda <jh...@gmail.com>]

First I would like to take this opportunity to thank this group for
releasing 0.8.2.0. It's a major milestone with a rich set of features.
Kudos to all the contributors! We are still running 0.8.1.2 and are
planning to upgrade to 0.8.2.0. While planning this upgrade we
discovered many topics that are no longer active and best be deleted.
Since it would be a common task faced by many kafka adopters I thought
I'd raise it here and seek expert advise. Basically what we desire is
a utility/script/steps to first identify the defunct topics (let's say
those topics that haven't seen any traffic in the past 'n' days) and
then delete them. Will appreciate your response.

Thanks,
Jagbir

Re: How to delete defunct topics

[Jagbir Hooda <jh...@gmail.com>]

Hi Joel,

The mbean approach depends upon storing traffic stats over a period of
time, so is doable but complex. I am thinking about another approach
to check logfiles time-stamp (to check producer activity) in
conjunction with the mtime for topic offsets (consumers activity). So
if both timestamp and mtime are older than 'n' days that may imply a
defunct topic. Will appreciate your feedback.

Thanks,
Jagbir

On Thu, Feb 5, 2015 at 6:57 PM, Joel Koshy <jj...@gmail.com> wrote:
> There are mbeans
> (http://kafka.apache.org/documentation.html#monitoring) that you can
> poke for incoming message rate - if you look at those over a period of
> time you can figure out which of those are likely to be defunct and
> then delete those topics.
>
> On Thu, Feb 05, 2015 at 02:38:27PM -0800, Jagbir Hooda wrote:
>> First I would like to take this opportunity to thank this group for
>> releasing 0.8.2.0. It's a major milestone with a rich set of features.
>> Kudos to all the contributors! We are still running 0.8.1.2 and are
>> planning to upgrade to 0.8.2.0. While planning this upgrade we
>> discovered many topics that are no longer active and best be deleted.
>> Since it would be a common task faced by many kafka adopters I thought
>> I'd raise it here and seek expert advise. Basically what we desire is
>> a utility/script/steps to first identify the defunct topics (let's say
>> those topics that haven't seen any traffic in the past 'n' days) and
>> then delete them. Will appreciate your response.
>>
>> Thanks,
>> Jagbir
>

Re: How to delete defunct topics

[Joel Koshy <jj...@gmail.com>]

There are mbeans
(http://kafka.apache.org/documentation.html#monitoring) that you can
poke for incoming message rate - if you look at those over a period of
time you can figure out which of those are likely to be defunct and
then delete those topics.

On Thu, Feb 05, 2015 at 02:38:27PM -0800, Jagbir Hooda wrote:
> First I would like to take this opportunity to thank this group for
> releasing 0.8.2.0. It's a major milestone with a rich set of features.
> Kudos to all the contributors! We are still running 0.8.1.2 and are
> planning to upgrade to 0.8.2.0. While planning this upgrade we
> discovered many topics that are no longer active and best be deleted.
> Since it would be a common task faced by many kafka adopters I thought
> I'd raise it here and seek expert advise. Basically what we desire is
> a utility/script/steps to first identify the defunct topics (let's say
> those topics that haven't seen any traffic in the past 'n' days) and
> then delete them. Will appreciate your response.
> 
> Thanks,
> Jagbir