You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Franco Giacosa <fg...@gmail.com> on 2016/02/23 11:28:10 UTC
problem deleting topic
Hello,
I am having the following problem trying to delete a topic.
The topic was auto-created with a default.replication.factor = 1, but my
test cluster has only 1 machine, so now when I start kafka I get this error
ERROR [KafkaApi-0] error when handling request Name: TopicMetadataRequest;
Version: 0; CorrelationId: 762828; ClientId:
adee54ba-d952-43a8-a19d-f05ce9a44997; Topics: abc1 (kafka.server.KafkaApis)
kafka.admin.AdminOperationException: replication factor: 1 larger than
available brokers: 0
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236)
at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:572)
at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:555)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at
scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:92)
at scala.collection.AbstractSet.map(Set.scala:47)
at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:555)
at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:624)
at kafka.server.KafkaApis.handle(KafkaApis.scala:71)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
I am trying to delete the topic (with delete.topic.enable=true), so when I
do
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic abc1
on kafka I can see all the INFO related to the topic been deleted.
But if I check the list of topics the topic is still listed and if I
restart kafka I get the same exception.
Ideas?
Thanks
Re: problem deleting topic
Posted by Leo Lin <le...@brigade.com>.
You can hard delete those topics by
1) Shutting down the cluster
2) Delete the topic information in zookeeper
3) Delete the physical logs in the kafka log dir (defined in
server.properties)
5) Update the checkpoint files in the log dir (remember to update the count
on line 2 for those files)
6) Restarting the cluster
On Wed, Feb 24, 2016 at 9:57 AM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> Hmmm... I don't know for sure, but any chance a re-boot of Zookeeper would
> help?
>
> Is your topic still in /admin/delete_topics? (On Zookeeper I mean)
>
> Also, how important is it to know what happened as opposed to just getting
> to a runnable state again?
>
> In other words, what time/effort will it cost to re-create your kafka
> topics from scratch? Is it worth it to troubleshoot?
>
> In my own testing recently, because I was on VMs and had control of the
> software that loaded the topics, I have wiped the Kafka VM several times,
> re-installed Kafka and re-done the configs (from my notes), then deleted
> everything related to Kafka in Zookeeper and run it all agin - but my cost
> for doing that was under two hours....
>
>
>
> On Wed, Feb 24, 2016 at 2:49 AM, Franco Giacosa <fg...@gmail.com>
> wrote:
>
> > Leo,
> >
> > I am running my broker in that machine, I get that error when I am
> starting
> > the broker.
> > I understand that when I delete the topic it takes a while, but this
> > happened to me yesterday while I was deleting other topics (that where
> > deleted successfully). But this 2 topics still throw those errors when I
> > start kafka.
> >
> > 2016-02-24 0:27 GMT+01:00 Leo Lin <le...@brigade.com>:
> >
> > > Hi Franco,
> > >
> > > The default.replication.factor shouldn't be a problem but the error
> > message
> > > does look like you don't have any available brokers?
> > > kafka.admin.AdminOperationException: replication factor: 1 larger than
> > > available brokers: 0
> > >
> > > When you are deleting the topic using the kafka topics tool it does not
> > > delete the topic instantly but it gets marked for deletion in
> zookeeper.
> > If
> > > you connect to your zookeeper cluster and look under
> > /admin/delete_topics,
> > > your deleted topic should be there. It takes awhile before Kafka
> actually
> > > deletes it.
> > >
> > > Here's some discussions around it
> > >
> > >
> >
> http://stackoverflow.com/questions/23976670/when-how-does-a-topic-marked-for-deletion-get-finally-removed
> > >
> > > Best,
> > > Leo
> > >
> > > On Tue, Feb 23, 2016 at 2:28 AM, Franco Giacosa <fg...@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I am having the following problem trying to delete a topic.
> > > >
> > > > The topic was auto-created with a default.replication.factor = 1, but
> > my
> > > > test cluster has only 1 machine, so now when I start kafka I get this
> > > error
> > > >
> > > > ERROR [KafkaApi-0] error when handling request Name:
> > > TopicMetadataRequest;
> > > > Version: 0; CorrelationId: 762828; ClientId:
> > > > adee54ba-d952-43a8-a19d-f05ce9a44997; Topics: abc1
> > > (kafka.server.KafkaApis)
> > > > kafka.admin.AdminOperationException: replication factor: 1 larger
> than
> > > > available brokers: 0
> > > > at
> kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77)
> > > > at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236)
> > > > at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:572)
> > > > at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:555)
> > > > at
> > > >
> > > >
> > >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> > > > at
> > > >
> > > >
> > >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> > > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
> > > > at
> > scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
> > > > at
> > > >
> > > >
> > >
> >
> scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
> > > > at scala.collection.SetLike$class.map(SetLike.scala:92)
> > > > at scala.collection.AbstractSet.map(Set.scala:47)
> > > > at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:555)
> > > > at
> > kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:624)
> > > > at kafka.server.KafkaApis.handle(KafkaApis.scala:71)
> > > > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > >
> > > >
> > > >
> > > > I am trying to delete the topic (with delete.topic.enable=true), so
> > when
> > > I
> > > > do
> > > >
> > > > bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic abc1
> > > >
> > > > on kafka I can see all the INFO related to the topic been deleted.
> > > >
> > > > But if I check the list of topics the topic is still listed and if I
> > > > restart kafka I get the same exception.
> > > >
> > > > Ideas?
> > > >
> > > > Thanks
> > > >
> > >
> > >
> > >
> > > --
> > > "Dream no small dreams for they have no power to move the hearts of
> men."
> > >
> > > Johann Wolfgang von Goethe
> > >
> >
>
--
"Dream no small dreams for they have no power to move the hearts of men."
Johann Wolfgang von Goethe
Re: problem deleting topic
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Hmmm... I don't know for sure, but any chance a re-boot of Zookeeper would
help?
Is your topic still in /admin/delete_topics? (On Zookeeper I mean)
Also, how important is it to know what happened as opposed to just getting
to a runnable state again?
In other words, what time/effort will it cost to re-create your kafka
topics from scratch? Is it worth it to troubleshoot?
In my own testing recently, because I was on VMs and had control of the
software that loaded the topics, I have wiped the Kafka VM several times,
re-installed Kafka and re-done the configs (from my notes), then deleted
everything related to Kafka in Zookeeper and run it all agin - but my cost
for doing that was under two hours....
On Wed, Feb 24, 2016 at 2:49 AM, Franco Giacosa <fg...@gmail.com> wrote:
> Leo,
>
> I am running my broker in that machine, I get that error when I am starting
> the broker.
> I understand that when I delete the topic it takes a while, but this
> happened to me yesterday while I was deleting other topics (that where
> deleted successfully). But this 2 topics still throw those errors when I
> start kafka.
>
> 2016-02-24 0:27 GMT+01:00 Leo Lin <le...@brigade.com>:
>
> > Hi Franco,
> >
> > The default.replication.factor shouldn't be a problem but the error
> message
> > does look like you don't have any available brokers?
> > kafka.admin.AdminOperationException: replication factor: 1 larger than
> > available brokers: 0
> >
> > When you are deleting the topic using the kafka topics tool it does not
> > delete the topic instantly but it gets marked for deletion in zookeeper.
> If
> > you connect to your zookeeper cluster and look under
> /admin/delete_topics,
> > your deleted topic should be there. It takes awhile before Kafka actually
> > deletes it.
> >
> > Here's some discussions around it
> >
> >
> http://stackoverflow.com/questions/23976670/when-how-does-a-topic-marked-for-deletion-get-finally-removed
> >
> > Best,
> > Leo
> >
> > On Tue, Feb 23, 2016 at 2:28 AM, Franco Giacosa <fg...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I am having the following problem trying to delete a topic.
> > >
> > > The topic was auto-created with a default.replication.factor = 1, but
> my
> > > test cluster has only 1 machine, so now when I start kafka I get this
> > error
> > >
> > > ERROR [KafkaApi-0] error when handling request Name:
> > TopicMetadataRequest;
> > > Version: 0; CorrelationId: 762828; ClientId:
> > > adee54ba-d952-43a8-a19d-f05ce9a44997; Topics: abc1
> > (kafka.server.KafkaApis)
> > > kafka.admin.AdminOperationException: replication factor: 1 larger than
> > > available brokers: 0
> > > at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77)
> > > at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236)
> > > at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:572)
> > > at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:555)
> > > at
> > >
> > >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> > > at
> > >
> > >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
> > > at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
> > > at
> > >
> > >
> >
> scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
> > > at scala.collection.SetLike$class.map(SetLike.scala:92)
> > > at scala.collection.AbstractSet.map(Set.scala:47)
> > > at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:555)
> > > at
> kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:624)
> > > at kafka.server.KafkaApis.handle(KafkaApis.scala:71)
> > > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > >
> > > I am trying to delete the topic (with delete.topic.enable=true), so
> when
> > I
> > > do
> > >
> > > bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic abc1
> > >
> > > on kafka I can see all the INFO related to the topic been deleted.
> > >
> > > But if I check the list of topics the topic is still listed and if I
> > > restart kafka I get the same exception.
> > >
> > > Ideas?
> > >
> > > Thanks
> > >
> >
> >
> >
> > --
> > "Dream no small dreams for they have no power to move the hearts of men."
> >
> > Johann Wolfgang von Goethe
> >
>
Re: problem deleting topic
Posted by Franco Giacosa <fg...@gmail.com>.
Leo,
I am running my broker in that machine, I get that error when I am starting
the broker.
I understand that when I delete the topic it takes a while, but this
happened to me yesterday while I was deleting other topics (that where
deleted successfully). But this 2 topics still throw those errors when I
start kafka.
2016-02-24 0:27 GMT+01:00 Leo Lin <le...@brigade.com>:
> Hi Franco,
>
> The default.replication.factor shouldn't be a problem but the error message
> does look like you don't have any available brokers?
> kafka.admin.AdminOperationException: replication factor: 1 larger than
> available brokers: 0
>
> When you are deleting the topic using the kafka topics tool it does not
> delete the topic instantly but it gets marked for deletion in zookeeper. If
> you connect to your zookeeper cluster and look under /admin/delete_topics,
> your deleted topic should be there. It takes awhile before Kafka actually
> deletes it.
>
> Here's some discussions around it
>
> http://stackoverflow.com/questions/23976670/when-how-does-a-topic-marked-for-deletion-get-finally-removed
>
> Best,
> Leo
>
> On Tue, Feb 23, 2016 at 2:28 AM, Franco Giacosa <fg...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I am having the following problem trying to delete a topic.
> >
> > The topic was auto-created with a default.replication.factor = 1, but my
> > test cluster has only 1 machine, so now when I start kafka I get this
> error
> >
> > ERROR [KafkaApi-0] error when handling request Name:
> TopicMetadataRequest;
> > Version: 0; CorrelationId: 762828; ClientId:
> > adee54ba-d952-43a8-a19d-f05ce9a44997; Topics: abc1
> (kafka.server.KafkaApis)
> > kafka.admin.AdminOperationException: replication factor: 1 larger than
> > available brokers: 0
> > at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77)
> > at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236)
> > at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:572)
> > at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:555)
> > at
> >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> > at
> >
> >
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> > at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
> > at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
> > at
> >
> >
> scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
> > at scala.collection.SetLike$class.map(SetLike.scala:92)
> > at scala.collection.AbstractSet.map(Set.scala:47)
> > at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:555)
> > at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:624)
> > at kafka.server.KafkaApis.handle(KafkaApis.scala:71)
> > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> > I am trying to delete the topic (with delete.topic.enable=true), so when
> I
> > do
> >
> > bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic abc1
> >
> > on kafka I can see all the INFO related to the topic been deleted.
> >
> > But if I check the list of topics the topic is still listed and if I
> > restart kafka I get the same exception.
> >
> > Ideas?
> >
> > Thanks
> >
>
>
>
> --
> "Dream no small dreams for they have no power to move the hearts of men."
>
> Johann Wolfgang von Goethe
>
Re: problem deleting topic
Posted by Leo Lin <le...@brigade.com>.
Hi Franco,
The default.replication.factor shouldn't be a problem but the error message
does look like you don't have any available brokers?
kafka.admin.AdminOperationException: replication factor: 1 larger than
available brokers: 0
When you are deleting the topic using the kafka topics tool it does not
delete the topic instantly but it gets marked for deletion in zookeeper. If
you connect to your zookeeper cluster and look under /admin/delete_topics,
your deleted topic should be there. It takes awhile before Kafka actually
deletes it.
Here's some discussions around it
http://stackoverflow.com/questions/23976670/when-how-does-a-topic-marked-for-deletion-get-finally-removed
Best,
Leo
On Tue, Feb 23, 2016 at 2:28 AM, Franco Giacosa <fg...@gmail.com> wrote:
> Hello,
>
> I am having the following problem trying to delete a topic.
>
> The topic was auto-created with a default.replication.factor = 1, but my
> test cluster has only 1 machine, so now when I start kafka I get this error
>
> ERROR [KafkaApi-0] error when handling request Name: TopicMetadataRequest;
> Version: 0; CorrelationId: 762828; ClientId:
> adee54ba-d952-43a8-a19d-f05ce9a44997; Topics: abc1 (kafka.server.KafkaApis)
> kafka.admin.AdminOperationException: replication factor: 1 larger than
> available brokers: 0
> at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77)
> at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236)
> at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:572)
> at kafka.server.KafkaApis$$anonfun$20.apply(KafkaApis.scala:555)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
> at
>
> scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
> at scala.collection.SetLike$class.map(SetLike.scala:92)
> at scala.collection.AbstractSet.map(Set.scala:47)
> at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:555)
> at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:624)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:71)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> I am trying to delete the topic (with delete.topic.enable=true), so when I
> do
>
> bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic abc1
>
> on kafka I can see all the INFO related to the topic been deleted.
>
> But if I check the list of topics the topic is still listed and if I
> restart kafka I get the same exception.
>
> Ideas?
>
> Thanks
>
--
"Dream no small dreams for they have no power to move the hearts of men."
Johann Wolfgang von Goethe