You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Luigi Tagliamonte <lu...@gmail.com> on 2018/08/21 23:24:22 UTC
Cleanup unused topics
I've got a sandbox cluster (without any control on topic creation) full of
unused topics and I'd like to cleanup all the those topics.
I was able to identify all the topics with 0 messages with the following
lines:
*#!/bin/bash*
*set -e*
*BROKERS="10.16.208.220:9092 <http://10.16.208.220:9092>,10.16.209.24:9092
<http://10.16.209.24:9092>,10.16.209.146:9092 <http://10.16.209.146:9092>"*
*ZOOKEEPER="10.16.209.44:2181 <http://10.16.209.44:2181>"*
*declare -a topics=($(/opt/kafka/bin/kafka-topics.sh --zookeeper
${ZOOKEEPER} --list | tr "\n" " "))*
*for topic in ${topics[@]}*
*do*
* messages=$(/opt/kafka/bin/kafka-run-class.sh
kafka.tools.GetOffsetShell --broker-list ${BROKERS} --topic $topic --time
-1| tr ":" " " | awk '{ sum += $3 } END { print sum }')*
* echo "$topic has $messages messages"*
*done*
but I'd like to sample also the topic usage, and do a match between the 2
lists.
Is there any JMX metric I can use to make sure nobody is receiving/creating
messages for each topic like a total counter or read/write per topic?
Cassandra has something like that so I was wondering if Kafka has something
like that as well.
Looking forward to your reply.
Regards
L.
Re: Cleanup unused topics
Posted by Luigi Tagliamonte <lu...@gmail.com>.
Thank you Brett I identified this metrics as well, thank you for confirming.
I ended up using TotalProduceRequestsPerSec and TotalFetchRequestsPerSec.
Regards
On Tue, Aug 21, 2018 at 9:53 PM Brett Rann <br...@zendesk.com.invalid>
wrote:
> These are in regex form for DataDog's JMX collector, but it should get you
> started:
>
> bean_regex:
>
> 'kafka\.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=.*'
> bean_regex:
>
> 'kafka\.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic=.*'
> bean_regex:
> 'kafka\.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=.*'
> bean_regex:
> 'kafka\.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=.*'
> bean_regex:
> 'kafka\.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=.*'
> bean_regex:
>
> 'kafka\.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec,topic=.*'
> bean_regex:
>
> 'kafka\.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec,topic=.*'
> bean_regex:
>
> 'kafka\.server:type=BrokerTopicMetrics,name=ProduceMessageConversionsPerSec,topic=.*'
> bean_regex:
>
> 'kafka\.server:type=BrokerTopicMetrics,name=FetchMessageConversionsPerSec,topic=.*'
>
> On Wed, Aug 22, 2018 at 9:24 AM Luigi Tagliamonte <
> luigi.tagliamonte86@gmail.com> wrote:
>
> > I've got a sandbox cluster (without any control on topic creation) full
> of
> > unused topics and I'd like to cleanup all the those topics.
> > I was able to identify all the topics with 0 messages with the following
> > lines:
> >
> > *#!/bin/bash*
> > *set -e*
> > *BROKERS="10.16.208.220:9092 <http://10.16.208.220:9092
> > <http://10.16.208.220:9092>>,
> > 10.16.209.24:9092
> > <http://10.16.209.24:9092
> > <http://10.16.209.24:9092>>,
> > 10.16.209.146:9092 <http://10.16.209.146:9092
> > <http://10.16.209.146:9092>>"*
> > *ZOOKEEPER="10.16.209.44:2181 <http://10.16.209.44:2181
> > <http://10.16.209.44:2181>>"*
> > *declare -a topics=($(/opt/kafka/bin/kafka-topics.sh --zookeeper
> > ${ZOOKEEPER} --list | tr "\n" " "))*
> > *for topic in ${topics[@]}*
> > *do*
> > * messages=$(/opt/kafka/bin/kafka-run-class.sh
> > kafka.tools.GetOffsetShell --broker-list ${BROKERS} --topic $topic --time
> > -1| tr ":" " " | awk '{ sum += $3 } END { print sum }')*
> > * echo "$topic has $messages messages"*
> > *done*
> >
> > but I'd like to sample also the topic usage, and do a match between the 2
> > lists.
> > Is there any JMX metric I can use to make sure nobody is
> receiving/creating
> > messages for each topic like a total counter or read/write per topic?
> > Cassandra has something like that so I was wondering if Kafka has
> something
> > like that as well.
> > Looking forward to your reply.
> > Regards
> > L.
> >
>
Re: Cleanup unused topics
Posted by Brett Rann <br...@zendesk.com.INVALID>.
These are in regex form for DataDog's JMX collector, but it should get you
started:
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=ProduceMessageConversionsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=FetchMessageConversionsPerSec,topic=.*'
On Wed, Aug 22, 2018 at 9:24 AM Luigi Tagliamonte <
luigi.tagliamonte86@gmail.com> wrote:
> I've got a sandbox cluster (without any control on topic creation) full of
> unused topics and I'd like to cleanup all the those topics.
> I was able to identify all the topics with 0 messages with the following
> lines:
>
> *#!/bin/bash*
> *set -e*
> *BROKERS="10.16.208.220:9092 <http://10.16.208.220:9092
> <http://10.16.208.220:9092>>,
> 10.16.209.24:9092
> <http://10.16.209.24:9092
> <http://10.16.209.24:9092>>,
> 10.16.209.146:9092 <http://10.16.209.146:9092
> <http://10.16.209.146:9092>>"*
> *ZOOKEEPER="10.16.209.44:2181 <http://10.16.209.44:2181
> <http://10.16.209.44:2181>>"*
> *declare -a topics=($(/opt/kafka/bin/kafka-topics.sh --zookeeper
> ${ZOOKEEPER} --list | tr "\n" " "))*
> *for topic in ${topics[@]}*
> *do*
> * messages=$(/opt/kafka/bin/kafka-run-class.sh
> kafka.tools.GetOffsetShell --broker-list ${BROKERS} --topic $topic --time
> -1| tr ":" " " | awk '{ sum += $3 } END { print sum }')*
> * echo "$topic has $messages messages"*
> *done*
>
> but I'd like to sample also the topic usage, and do a match between the 2
> lists.
> Is there any JMX metric I can use to make sure nobody is receiving/creating
> messages for each topic like a total counter or read/write per topic?
> Cassandra has something like that so I was wondering if Kafka has something
> like that as well.
> Looking forward to your reply.
> Regards
> L.
>