You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Rakesh Vidyadharan <rv...@gracenote.com> on 2015/12/03 23:30:36 UTC
Failed attempt to delete topic
Hello,
We are on an older kafka (0.8.1) version. While a number of consumers were running, we attempted to delete a few topics using the kafka-topics.sh file (basically want to remove all messages in that topic and restart, since our entities went through some incompatible changes). We noticed logs saying the topic has been queued for deletion. After stopping all processes accessing kafka, we restarted kafka and then our processes. The old topics do not seem to have been deleted (I can still see the log directories corresponding to the topics), and none of the clients are able to either publish or read to the topics that we attempted to delete. Attempting to read gives us the following type of error:
Attempting to access an invalid KafkaTopic ( are you operating on a closed KafkaTopic ?)
Attempting to publish gives us a more general type of error:
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:76)
How can be get around this issue and start using the topics that we tried to clean up? There may have been better ways to achieve what we wanted, if so please suggest recommendations as well.
Thanks
Rakesh
Re: Failed attempt to delete topic
Posted by Rakesh Vidyadharan <rv...@gracenote.com>.
Hi Mayuresh
These are some of the relevant logs that I could find
[2015-12-03 16:04:23,594] INFO Loading log 'merckx.raw.event.type-0' (kafka.log.LogManager)
[2015-12-03 16:04:23,595] INFO Completed load of log merckx.raw.event.type-0 with log end offset 2 (kafka.log.Log)
[2015-12-03 16:04:23,641] INFO Loading log 'merckx.raw.event-0' (kafka.log.LogManager)
[2015-12-03 16:04:23,643] INFO Completed load of log merckx.raw.event-0 with log end offset 25149 (kafka.log.Log)
[2015-12-03 16:04:25,208] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [merckx.raw.venue.capacity,0],[merckx.raw.sport.franchise,0],[merckx.raw.college,0],[merckx.raw.venue.formerName,0],[merckx.raw.venue,0],[merckx.raw.event,0],[merckx.raw.organization,0],[merckx.raw.sport.franchise.season,0],[merckx.raw.organization.season,0],[merckx.raw.sport,0],[merckx.raw.event.type,0] (kafka.server.ReplicaFetcherManager)
[2015-12-03 16:04:25,279] INFO Truncating log merckx.raw.event-0 to offset 25149. (kafka.log.Log)
[2015-12-03 16:04:25,280] INFO Truncating log merckx.raw.event.type-0 to offset 2. (kafka.log.Log)
[2015-12-03 16:09:04,709] ERROR [KafkaApi-0] Error while fetching metadata for partition [merckx.raw.event,0] (kafka.server.KafkaApis)
kafka.common.LeaderNotAvailableException: Leader not available for partition [merckx.raw.event,0]
[2015-12-03 16:09:05,926] ERROR [KafkaApi-0] Error while fetching metadata for partition [merckx.raw.event,0] (kafka.server.KafkaApis)
Thanks
Rakesh
On 03/12/2015 17:33, "Mayuresh Gharat" <gh...@gmail.com> wrote:
>Can you paste some logs from the controller, when you deleted the topic?
>
>Thanks,
>
>Mayuresh
>
>On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
>rvidyadharan@gracenote.com> wrote:
>
>> Hello,
>>
>> We are on an older kafka (0.8.1) version. While a number of consumers
>> were running, we attempted to delete a few topics using the kafka-topics.sh
>> file (basically want to remove all messages in that topic and restart,
>> since our entities went through some incompatible changes). We noticed
>> logs saying the topic has been queued for deletion. After stopping all
>> processes accessing kafka, we restarted kafka and then our processes. The
>> old topics do not seem to have been deleted (I can still see the log
>> directories corresponding to the topics), and none of the clients are able
>> to either publish or read to the topics that we attempted to delete.
>> Attempting to read gives us the following type of error:
>>
>> Attempting to access an invalid KafkaTopic ( are you operating on a closed
>> KafkaTopic ?)
>>
>> Attempting to publish gives us a more general type of error:
>>
>> kafka.common.FailedToSendMessageException: Failed to send messages after 3
>> tries.
>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>> at kafka.producer.Producer.send(Producer.scala:76)
>>
>> How can be get around this issue and start using the topics that we tried
>> to clean up? There may have been better ways to achieve what we wanted, if
>> so please suggest recommendations as well.
>>
>> Thanks
>> Rakesh
>>
>>
>
>
>--
>-Regards,
>Mayuresh R. Gharat
>(862) 250-7125
Re: Failed attempt to delete topic
Posted by Steve Robenalt <sr...@highwire.org>.
Hi Rakesh,
Topic deletion didn't really work properly until 0.8.2. Here's a
stackoverflow link that summarizes how to work around this limitation:
http://stackoverflow.com/questions/24287900/delete-topic-in-kafka-0-8-1-1
HTH,
Steve
On Thu, Dec 3, 2015 at 3:33 PM, Mayuresh Gharat <gh...@gmail.com>
wrote:
> Can you paste some logs from the controller, when you deleted the topic?
>
> Thanks,
>
> Mayuresh
>
> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
> rvidyadharan@gracenote.com> wrote:
>
> > Hello,
> >
> > We are on an older kafka (0.8.1) version. While a number of consumers
> > were running, we attempted to delete a few topics using the
> kafka-topics.sh
> > file (basically want to remove all messages in that topic and restart,
> > since our entities went through some incompatible changes). We noticed
> > logs saying the topic has been queued for deletion. After stopping all
> > processes accessing kafka, we restarted kafka and then our processes.
> The
> > old topics do not seem to have been deleted (I can still see the log
> > directories corresponding to the topics), and none of the clients are
> able
> > to either publish or read to the topics that we attempted to delete.
> > Attempting to read gives us the following type of error:
> >
> > Attempting to access an invalid KafkaTopic ( are you operating on a
> closed
> > KafkaTopic ?)
> >
> > Attempting to publish gives us a more general type of error:
> >
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> > at
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > at kafka.producer.Producer.send(Producer.scala:76)
> >
> > How can be get around this issue and start using the topics that we tried
> > to clean up? There may have been better ways to achieve what we wanted,
> if
> > so please suggest recommendations as well.
> >
> > Thanks
> > Rakesh
> >
> >
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
--
Steve Robenalt
Software Architect
srobenalt@highwire.org <bz...@highwire.org>
(office/cell): 916-505-1785
HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org
Technology for Scholarly Communication
Re: Failed attempt to delete topic
Posted by Rakesh Vidyadharan <rv...@gracenote.com>.
Thanks Mayuresh. I was able to use the shell to delete the entries and things are working fine now.
On 03/12/2015 18:22, "Mayuresh Gharat" <gh...@gmail.com> wrote:
>you can use the zookeeper shell inside the bin directory for that.
>
>Thanks,
>
>Mayuresh
>
>On Thu, Dec 3, 2015 at 4:04 PM, Rakesh Vidyadharan <
>rvidyadharan@gracenote.com> wrote:
>
>> Thanks Stevo. I did see some messages related to /admin/delete_topics.
>> Will do some research on how I can clean up zookeeper.
>>
>> Thanks
>> Rakesh
>>
>>
>>
>>
>> On 03/12/2015 17:55, "Stevo Slavić" <ss...@gmail.com> wrote:
>>
>> >Delete was actually considered to be working since Kafka 0.8.2 (although
>> >there are still not easily reproducible edge cases when it doesn't work
>> >well even in in 0.8.2 or newer).
>> >In 0.8.1 one could request topic to be deleted (request gets stored as
>> >entry in ZooKeeper), because of presence of the request for topic to be
>> >deleted topic would become unusable (cannot publish or read), but broker
>> >would actually never (work on the request to) delete topic.
>> >
>> >Maybe it will be enough to delete from ZooKeeper entry for the topic
>> >deletion request under /admin/delete_topics to have topic usable again.
>> >
>> >Otherwise, just upgrade broker side to 0.8.2.x or latest 0.9.0.0 - new
>> >broker should work with old clients so maybe you don't have to upgrade
>> >client side immediately.
>> >
>> >Kind regards,
>> >Stevo Slavic.
>> >
>> >
>> >On Fri, Dec 4, 2015 at 12:33 AM, Mayuresh Gharat <
>> gharatmayuresh15@gmail.com
>> >> wrote:
>> >
>> >> Can you paste some logs from the controller, when you deleted the topic?
>> >>
>> >> Thanks,
>> >>
>> >> Mayuresh
>> >>
>> >> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
>> >> rvidyadharan@gracenote.com> wrote:
>> >>
>> >> > Hello,
>> >> >
>> >> > We are on an older kafka (0.8.1) version. While a number of consumers
>> >> > were running, we attempted to delete a few topics using the
>> >> kafka-topics.sh
>> >> > file (basically want to remove all messages in that topic and restart,
>> >> > since our entities went through some incompatible changes). We
>> noticed
>> >> > logs saying the topic has been queued for deletion. After stopping
>> all
>> >> > processes accessing kafka, we restarted kafka and then our processes.
>> >> The
>> >> > old topics do not seem to have been deleted (I can still see the log
>> >> > directories corresponding to the topics), and none of the clients are
>> >> able
>> >> > to either publish or read to the topics that we attempted to delete.
>> >> > Attempting to read gives us the following type of error:
>> >> >
>> >> > Attempting to access an invalid KafkaTopic ( are you operating on a
>> >> closed
>> >> > KafkaTopic ?)
>> >> >
>> >> > Attempting to publish gives us a more general type of error:
>> >> >
>> >> > kafka.common.FailedToSendMessageException: Failed to send messages
>> after
>> >> 3
>> >> > tries.
>> >> > at
>> >> >
>> >>
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>> >> > at kafka.producer.Producer.send(Producer.scala:76)
>> >> >
>> >> > How can be get around this issue and start using the topics that we
>> tried
>> >> > to clean up? There may have been better ways to achieve what we
>> wanted,
>> >> if
>> >> > so please suggest recommendations as well.
>> >> >
>> >> > Thanks
>> >> > Rakesh
>> >> >
>> >> >
>> >>
>> >>
>> >> --
>> >> -Regards,
>> >> Mayuresh R. Gharat
>> >> (862) 250-7125
>> >>
>>
>
>
>
>--
>-Regards,
>Mayuresh R. Gharat
>(862) 250-7125
Re: Failed attempt to delete topic
Posted by Mayuresh Gharat <gh...@gmail.com>.
you can use the zookeeper shell inside the bin directory for that.
Thanks,
Mayuresh
On Thu, Dec 3, 2015 at 4:04 PM, Rakesh Vidyadharan <
rvidyadharan@gracenote.com> wrote:
> Thanks Stevo. I did see some messages related to /admin/delete_topics.
> Will do some research on how I can clean up zookeeper.
>
> Thanks
> Rakesh
>
>
>
>
> On 03/12/2015 17:55, "Stevo Slavić" <ss...@gmail.com> wrote:
>
> >Delete was actually considered to be working since Kafka 0.8.2 (although
> >there are still not easily reproducible edge cases when it doesn't work
> >well even in in 0.8.2 or newer).
> >In 0.8.1 one could request topic to be deleted (request gets stored as
> >entry in ZooKeeper), because of presence of the request for topic to be
> >deleted topic would become unusable (cannot publish or read), but broker
> >would actually never (work on the request to) delete topic.
> >
> >Maybe it will be enough to delete from ZooKeeper entry for the topic
> >deletion request under /admin/delete_topics to have topic usable again.
> >
> >Otherwise, just upgrade broker side to 0.8.2.x or latest 0.9.0.0 - new
> >broker should work with old clients so maybe you don't have to upgrade
> >client side immediately.
> >
> >Kind regards,
> >Stevo Slavic.
> >
> >
> >On Fri, Dec 4, 2015 at 12:33 AM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com
> >> wrote:
> >
> >> Can you paste some logs from the controller, when you deleted the topic?
> >>
> >> Thanks,
> >>
> >> Mayuresh
> >>
> >> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
> >> rvidyadharan@gracenote.com> wrote:
> >>
> >> > Hello,
> >> >
> >> > We are on an older kafka (0.8.1) version. While a number of consumers
> >> > were running, we attempted to delete a few topics using the
> >> kafka-topics.sh
> >> > file (basically want to remove all messages in that topic and restart,
> >> > since our entities went through some incompatible changes). We
> noticed
> >> > logs saying the topic has been queued for deletion. After stopping
> all
> >> > processes accessing kafka, we restarted kafka and then our processes.
> >> The
> >> > old topics do not seem to have been deleted (I can still see the log
> >> > directories corresponding to the topics), and none of the clients are
> >> able
> >> > to either publish or read to the topics that we attempted to delete.
> >> > Attempting to read gives us the following type of error:
> >> >
> >> > Attempting to access an invalid KafkaTopic ( are you operating on a
> >> closed
> >> > KafkaTopic ?)
> >> >
> >> > Attempting to publish gives us a more general type of error:
> >> >
> >> > kafka.common.FailedToSendMessageException: Failed to send messages
> after
> >> 3
> >> > tries.
> >> > at
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> >> > at kafka.producer.Producer.send(Producer.scala:76)
> >> >
> >> > How can be get around this issue and start using the topics that we
> tried
> >> > to clean up? There may have been better ways to achieve what we
> wanted,
> >> if
> >> > so please suggest recommendations as well.
> >> >
> >> > Thanks
> >> > Rakesh
> >> >
> >> >
> >>
> >>
> >> --
> >> -Regards,
> >> Mayuresh R. Gharat
> >> (862) 250-7125
> >>
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Failed attempt to delete topic
Posted by Rakesh Vidyadharan <rv...@gracenote.com>.
Thanks Stevo. I did see some messages related to /admin/delete_topics. Will do some research on how I can clean up zookeeper.
Thanks
Rakesh
On 03/12/2015 17:55, "Stevo Slavić" <ss...@gmail.com> wrote:
>Delete was actually considered to be working since Kafka 0.8.2 (although
>there are still not easily reproducible edge cases when it doesn't work
>well even in in 0.8.2 or newer).
>In 0.8.1 one could request topic to be deleted (request gets stored as
>entry in ZooKeeper), because of presence of the request for topic to be
>deleted topic would become unusable (cannot publish or read), but broker
>would actually never (work on the request to) delete topic.
>
>Maybe it will be enough to delete from ZooKeeper entry for the topic
>deletion request under /admin/delete_topics to have topic usable again.
>
>Otherwise, just upgrade broker side to 0.8.2.x or latest 0.9.0.0 - new
>broker should work with old clients so maybe you don't have to upgrade
>client side immediately.
>
>Kind regards,
>Stevo Slavic.
>
>
>On Fri, Dec 4, 2015 at 12:33 AM, Mayuresh Gharat <gharatmayuresh15@gmail.com
>> wrote:
>
>> Can you paste some logs from the controller, when you deleted the topic?
>>
>> Thanks,
>>
>> Mayuresh
>>
>> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
>> rvidyadharan@gracenote.com> wrote:
>>
>> > Hello,
>> >
>> > We are on an older kafka (0.8.1) version. While a number of consumers
>> > were running, we attempted to delete a few topics using the
>> kafka-topics.sh
>> > file (basically want to remove all messages in that topic and restart,
>> > since our entities went through some incompatible changes). We noticed
>> > logs saying the topic has been queued for deletion. After stopping all
>> > processes accessing kafka, we restarted kafka and then our processes.
>> The
>> > old topics do not seem to have been deleted (I can still see the log
>> > directories corresponding to the topics), and none of the clients are
>> able
>> > to either publish or read to the topics that we attempted to delete.
>> > Attempting to read gives us the following type of error:
>> >
>> > Attempting to access an invalid KafkaTopic ( are you operating on a
>> closed
>> > KafkaTopic ?)
>> >
>> > Attempting to publish gives us a more general type of error:
>> >
>> > kafka.common.FailedToSendMessageException: Failed to send messages after
>> 3
>> > tries.
>> > at
>> >
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>> > at kafka.producer.Producer.send(Producer.scala:76)
>> >
>> > How can be get around this issue and start using the topics that we tried
>> > to clean up? There may have been better ways to achieve what we wanted,
>> if
>> > so please suggest recommendations as well.
>> >
>> > Thanks
>> > Rakesh
>> >
>> >
>>
>>
>> --
>> -Regards,
>> Mayuresh R. Gharat
>> (862) 250-7125
>>
Re: Failed attempt to delete topic
Posted by Stevo Slavić <ss...@gmail.com>.
Delete was actually considered to be working since Kafka 0.8.2 (although
there are still not easily reproducible edge cases when it doesn't work
well even in in 0.8.2 or newer).
In 0.8.1 one could request topic to be deleted (request gets stored as
entry in ZooKeeper), because of presence of the request for topic to be
deleted topic would become unusable (cannot publish or read), but broker
would actually never (work on the request to) delete topic.
Maybe it will be enough to delete from ZooKeeper entry for the topic
deletion request under /admin/delete_topics to have topic usable again.
Otherwise, just upgrade broker side to 0.8.2.x or latest 0.9.0.0 - new
broker should work with old clients so maybe you don't have to upgrade
client side immediately.
Kind regards,
Stevo Slavic.
On Fri, Dec 4, 2015 at 12:33 AM, Mayuresh Gharat <gharatmayuresh15@gmail.com
> wrote:
> Can you paste some logs from the controller, when you deleted the topic?
>
> Thanks,
>
> Mayuresh
>
> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
> rvidyadharan@gracenote.com> wrote:
>
> > Hello,
> >
> > We are on an older kafka (0.8.1) version. While a number of consumers
> > were running, we attempted to delete a few topics using the
> kafka-topics.sh
> > file (basically want to remove all messages in that topic and restart,
> > since our entities went through some incompatible changes). We noticed
> > logs saying the topic has been queued for deletion. After stopping all
> > processes accessing kafka, we restarted kafka and then our processes.
> The
> > old topics do not seem to have been deleted (I can still see the log
> > directories corresponding to the topics), and none of the clients are
> able
> > to either publish or read to the topics that we attempted to delete.
> > Attempting to read gives us the following type of error:
> >
> > Attempting to access an invalid KafkaTopic ( are you operating on a
> closed
> > KafkaTopic ?)
> >
> > Attempting to publish gives us a more general type of error:
> >
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> > at
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > at kafka.producer.Producer.send(Producer.scala:76)
> >
> > How can be get around this issue and start using the topics that we tried
> > to clean up? There may have been better ways to achieve what we wanted,
> if
> > so please suggest recommendations as well.
> >
> > Thanks
> > Rakesh
> >
> >
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Failed attempt to delete topic
Posted by Mayuresh Gharat <gh...@gmail.com>.
Can you paste some logs from the controller, when you deleted the topic?
Thanks,
Mayuresh
On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
rvidyadharan@gracenote.com> wrote:
> Hello,
>
> We are on an older kafka (0.8.1) version. While a number of consumers
> were running, we attempted to delete a few topics using the kafka-topics.sh
> file (basically want to remove all messages in that topic and restart,
> since our entities went through some incompatible changes). We noticed
> logs saying the topic has been queued for deletion. After stopping all
> processes accessing kafka, we restarted kafka and then our processes. The
> old topics do not seem to have been deleted (I can still see the log
> directories corresponding to the topics), and none of the clients are able
> to either publish or read to the topics that we attempted to delete.
> Attempting to read gives us the following type of error:
>
> Attempting to access an invalid KafkaTopic ( are you operating on a closed
> KafkaTopic ?)
>
> Attempting to publish gives us a more general type of error:
>
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at kafka.producer.Producer.send(Producer.scala:76)
>
> How can be get around this issue and start using the topics that we tried
> to clean up? There may have been better ways to achieve what we wanted, if
> so please suggest recommendations as well.
>
> Thanks
> Rakesh
>
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125