You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Adrian McCague <Ad...@zopa.com> on 2017/04/06 17:33:12 UTC
Topic deletion
Hi all,
I am trying to understand topic deletion in kafka, there appears to be very little documentation or answers on how this works. Typically they just say to turn on the feature on the broker (in my case it is).
I executed:
Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
Running this again yields:
Topic mytopic is already marked for deletion.
--describe yields:
Topic: mytopic PartitionCount:6 ReplicationFactor:3 Configs:retention.ms=0
Topic: mytopic Partition: 0 Leader: -1 Replicas: 1006,1001,1005 Isr:
Topic mytopic Partition: 1 Leader: -1 Replicas: 1001,1005,1003 Isr:
Topic: mytopic Partition: 2 Leader: -1 Replicas: 1005,1003,1004 Isr:
Topic: mytopic Partition: 3 Leader: -1 Replicas: 1003,1004,1007 Isr:
Topic: mytopic Partition: 4 Leader: -1 Replicas: 1004,1007,1006 Isr:
Topic: mytopic Partition: 5 Leader: -1 Replicas: 1007,1006,1001 Isr:
You can see that the deletion mark has meant that the Leader is -1.
Also I read somewhere that retention needs to be set to something low to trigger the deletion, hence the config of retention.ms=0
Consumers (or streams in my case) no longer see the topic:
org.apache.kafka.streams.errors.TopologyBuilderException: Invalid topology building: stream-thread [StreamThread-1] Topic not found: mytopic
And I can't create a new topic in its place:
[2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
I am a little lost as to where to go next, could someone explain how topic deletion is actually applied when a topic is 'marked' for deletion as that may help trigger it.
Thanks!
Adrian
RE: Topic deletion
Posted by Adrian McCague <Ad...@zopa.com>.
Hi Zakee, this is 0.10.2.0. Thanks for heads up on that issue.
-----Original Message-----
From: Zakee [mailto:kzakee1@netzero.net]
Sent: 06 April 2017 19:28
To: users@kafka.apache.org
Subject: Re: Topic deletion
What version of kafka you are on? There are issues with delete topic until I guess kafka 0.10.1.0
You may be hitting this issue… https://issues.apache.org/jira/browse/KAFKA-2231 <https://issues.apache.org/jira/browse/KAFKA-2231>
Thanks
Zakee
> On Apr 6, 2017, at 11:20 AM, Adrian McCague <Ad...@zopa.com> wrote:
>
> Hi Sachin,
>
> I am told with confidence that the setting is present in server.properties (I do not have access to the server).
>
> I have used delete before without issue, and attempted to use it when the setting is false and the behaviour is different. I'm certain that running the command multiple times always shows the warning, that without the setting being true, the command has no effect. In this case it confirms that the topic is marked for deletion.
>
> Thanks
> Adrian
>
> -----Original Message-----
> From: Sachin Mittal [mailto:sjmittal@gmail.com]
> Sent: 06 April 2017 19:05
> To: users@kafka.apache.org
> Subject: Re: Topic deletion
>
> Do you have delete.topic.enable=true uncommented or present in
> server.properties
>
> On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague
> <Ad...@zopa.com>
> wrote:
>
>> Hi all,
>>
>> I am trying to understand topic deletion in kafka, there appears to
>> be very little documentation or answers on how this works. Typically
>> they just say to turn on the feature on the broker (in my case it is).
>>
>> I executed:
>> Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
>>
>> Running this again yields:
>> Topic mytopic is already marked for deletion.
>>
>> --describe yields:
>> Topic: mytopic PartitionCount:6 ReplicationFactor:3 Configs:
>> retention.ms=0
>> Topic: mytopic Partition: 0 Leader: -1 Replicas:
>> 1006,1001,1005 Isr:
>> Topic mytopic Partition: 1 Leader: -1 Replicas:
>> 1001,1005,1003 Isr:
>> Topic: mytopic Partition: 2 Leader: -1 Replicas:
>> 1005,1003,1004 Isr:
>> Topic: mytopic Partition: 3 Leader: -1 Replicas:
>> 1003,1004,1007 Isr:
>> Topic: mytopic Partition: 4 Leader: -1 Replicas:
>> 1004,1007,1006 Isr:
>> Topic: mytopic Partition: 5 Leader: -1 Replicas:
>> 1007,1006,1001 Isr:
>>
>> You can see that the deletion mark has meant that the Leader is -1.
>> Also I read somewhere that retention needs to be set to something low
>> to trigger the deletion, hence the config of retention.ms=0
>>
>> Consumers (or streams in my case) no longer see the topic:
>> org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
>> topology building: stream-thread [StreamThread-1] Topic not found:
>> mytopic
>>
>> And I can't create a new topic in its place:
>> [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.errors.TopicExistsException:
>> Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
>>
>> I am a little lost as to where to go next, could someone explain how
>> topic deletion is actually applied when a topic is 'marked' for
>> deletion as that may help trigger it.
>>
>> Thanks!
>> Adrian
>>
>>
____________________________________________________________
Affordable Wireless Plans
Set up is easy. Get online in minutes.
Starting at only $14.95 per month!
www.netzero.net?refcd=nzmem0216
Re: Topic deletion
Posted by Zakee <kz...@netzero.net>.
What version of kafka you are on? There are issues with delete topic until I guess kafka 0.10.1.0
You may be hitting this issue… https://issues.apache.org/jira/browse/KAFKA-2231 <https://issues.apache.org/jira/browse/KAFKA-2231>
Thanks
Zakee
> On Apr 6, 2017, at 11:20 AM, Adrian McCague <Ad...@zopa.com> wrote:
>
> Hi Sachin,
>
> I am told with confidence that the setting is present in server.properties (I do not have access to the server).
>
> I have used delete before without issue, and attempted to use it when the setting is false and the behaviour is different. I'm certain that running the command multiple times always shows the warning, that without the setting being true, the command has no effect. In this case it confirms that the topic is marked for deletion.
>
> Thanks
> Adrian
>
> -----Original Message-----
> From: Sachin Mittal [mailto:sjmittal@gmail.com]
> Sent: 06 April 2017 19:05
> To: users@kafka.apache.org
> Subject: Re: Topic deletion
>
> Do you have delete.topic.enable=true uncommented or present in server.properties
>
> On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague <Ad...@zopa.com>
> wrote:
>
>> Hi all,
>>
>> I am trying to understand topic deletion in kafka, there appears to be
>> very little documentation or answers on how this works. Typically they
>> just say to turn on the feature on the broker (in my case it is).
>>
>> I executed:
>> Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
>>
>> Running this again yields:
>> Topic mytopic is already marked for deletion.
>>
>> --describe yields:
>> Topic: mytopic PartitionCount:6 ReplicationFactor:3 Configs:
>> retention.ms=0
>> Topic: mytopic Partition: 0 Leader: -1 Replicas:
>> 1006,1001,1005 Isr:
>> Topic mytopic Partition: 1 Leader: -1 Replicas:
>> 1001,1005,1003 Isr:
>> Topic: mytopic Partition: 2 Leader: -1 Replicas:
>> 1005,1003,1004 Isr:
>> Topic: mytopic Partition: 3 Leader: -1 Replicas:
>> 1003,1004,1007 Isr:
>> Topic: mytopic Partition: 4 Leader: -1 Replicas:
>> 1004,1007,1006 Isr:
>> Topic: mytopic Partition: 5 Leader: -1 Replicas:
>> 1007,1006,1001 Isr:
>>
>> You can see that the deletion mark has meant that the Leader is -1.
>> Also I read somewhere that retention needs to be set to something low
>> to trigger the deletion, hence the config of retention.ms=0
>>
>> Consumers (or streams in my case) no longer see the topic:
>> org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
>> topology building: stream-thread [StreamThread-1] Topic not found:
>> mytopic
>>
>> And I can't create a new topic in its place:
>> [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.errors.TopicExistsException:
>> Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
>>
>> I am a little lost as to where to go next, could someone explain how
>> topic deletion is actually applied when a topic is 'marked' for
>> deletion as that may help trigger it.
>>
>> Thanks!
>> Adrian
>>
>>
____________________________________________________________
Affordable Wireless Plans
Set up is easy. Get online in minutes.
Starting at only $14.95 per month!
www.netzero.net?refcd=nzmem0216
RE: Topic deletion
Posted by Adrian McCague <Ad...@zopa.com>.
Hi Sachin,
I am told with confidence that the setting is present in server.properties (I do not have access to the server).
I have used delete before without issue, and attempted to use it when the setting is false and the behaviour is different. I'm certain that running the command multiple times always shows the warning, that without the setting being true, the command has no effect. In this case it confirms that the topic is marked for deletion.
Thanks
Adrian
-----Original Message-----
From: Sachin Mittal [mailto:sjmittal@gmail.com]
Sent: 06 April 2017 19:05
To: users@kafka.apache.org
Subject: Re: Topic deletion
Do you have delete.topic.enable=true uncommented or present in server.properties
On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague <Ad...@zopa.com>
wrote:
> Hi all,
>
> I am trying to understand topic deletion in kafka, there appears to be
> very little documentation or answers on how this works. Typically they
> just say to turn on the feature on the broker (in my case it is).
>
> I executed:
> Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
>
> Running this again yields:
> Topic mytopic is already marked for deletion.
>
> --describe yields:
> Topic: mytopic PartitionCount:6 ReplicationFactor:3 Configs:
> retention.ms=0
> Topic: mytopic Partition: 0 Leader: -1 Replicas:
> 1006,1001,1005 Isr:
> Topic mytopic Partition: 1 Leader: -1 Replicas:
> 1001,1005,1003 Isr:
> Topic: mytopic Partition: 2 Leader: -1 Replicas:
> 1005,1003,1004 Isr:
> Topic: mytopic Partition: 3 Leader: -1 Replicas:
> 1003,1004,1007 Isr:
> Topic: mytopic Partition: 4 Leader: -1 Replicas:
> 1004,1007,1006 Isr:
> Topic: mytopic Partition: 5 Leader: -1 Replicas:
> 1007,1006,1001 Isr:
>
> You can see that the deletion mark has meant that the Leader is -1.
> Also I read somewhere that retention needs to be set to something low
> to trigger the deletion, hence the config of retention.ms=0
>
> Consumers (or streams in my case) no longer see the topic:
> org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
> topology building: stream-thread [StreamThread-1] Topic not found:
> mytopic
>
> And I can't create a new topic in its place:
> [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.errors.TopicExistsException:
> Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
>
> I am a little lost as to where to go next, could someone explain how
> topic deletion is actually applied when a topic is 'marked' for
> deletion as that may help trigger it.
>
> Thanks!
> Adrian
>
>
Re: Topic deletion
Posted by Sachin Mittal <sj...@gmail.com>.
Do you have delete.topic.enable=true uncommented or present in
server.properties
On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague <Ad...@zopa.com>
wrote:
> Hi all,
>
> I am trying to understand topic deletion in kafka, there appears to be
> very little documentation or answers on how this works. Typically they just
> say to turn on the feature on the broker (in my case it is).
>
> I executed:
> Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
>
> Running this again yields:
> Topic mytopic is already marked for deletion.
>
> --describe yields:
> Topic: mytopic PartitionCount:6 ReplicationFactor:3 Configs:
> retention.ms=0
> Topic: mytopic Partition: 0 Leader: -1 Replicas:
> 1006,1001,1005 Isr:
> Topic mytopic Partition: 1 Leader: -1 Replicas:
> 1001,1005,1003 Isr:
> Topic: mytopic Partition: 2 Leader: -1 Replicas:
> 1005,1003,1004 Isr:
> Topic: mytopic Partition: 3 Leader: -1 Replicas:
> 1003,1004,1007 Isr:
> Topic: mytopic Partition: 4 Leader: -1 Replicas:
> 1004,1007,1006 Isr:
> Topic: mytopic Partition: 5 Leader: -1 Replicas:
> 1007,1006,1001 Isr:
>
> You can see that the deletion mark has meant that the Leader is -1.
> Also I read somewhere that retention needs to be set to something low to
> trigger the deletion, hence the config of retention.ms=0
>
> Consumers (or streams in my case) no longer see the topic:
> org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
> topology building: stream-thread [StreamThread-1] Topic not found: mytopic
>
> And I can't create a new topic in its place:
> [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.errors.TopicExistsException:
> Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
>
> I am a little lost as to where to go next, could someone explain how topic
> deletion is actually applied when a topic is 'marked' for deletion as that
> may help trigger it.
>
> Thanks!
> Adrian
>
>
RE: Topic deletion
Posted by Adrian McCague <Ad...@zopa.com>.
Thank you Mayuresh, that's an excellent write-up.
As you mention about reassignment blocking deletion; I think we are seeing an issue with our brokers and reassignment so that could well be the root cause of the deletion issue we experienced. The zKclient suggestion that I followed before resolved the issue on ZK but the brokers never resolved so we rebuilt the system (luckily this is in a staging environment).
Thanks
Adrian
-----Original Message-----
From: Mayuresh Gharat [mailto:gharatmayuresh15@gmail.com]
Sent: 07 April 2017 18:56
To: users@kafka.apache.org
Subject: Re: Topic deletion
Hi Adrian,
When you delete a topic, it is marked a under /admin/delete_topics path in zookeeper. Once that is done, controller gets a notification that a topic is marked for delete. The controller then notifies the replcias of the topic-partitions to delete the data for the topic and the topic itself.
After KAFKA-1911, delete topic is made asynchronous and faster. The replicas on receiving the notification from controller will rename the topic-partition directory to something like topic-partition.delete and return to controller saying delete topic is done. The controller when it receives the ack from all the replicas that topic has been deleted, it will delete the znode under /admin/delete_topics path. Your delete topic is successful at this stage. The renamed directory gets removed asynchronously on those replicas brokers at some later point in time.
Also remember, if you topic-partitions are been reassigned, delete topic will not be triggered till the reassignment is finished.
We at Linkedin, have been suing Delete topic for a while now and it has been working fine after KAFKA-1911.
Hope this helps.
Thanks,
Mayuresh
On Fri, Apr 7, 2017 at 2:15 AM, Adrian McCague <Ad...@zopa.com>
wrote:
> Indeed, I wish it was explained somewhere what marked for deletion
> does, and what process handles it. I will have to go and investigate the source.
>
> I can confirm the zkCli did the trick, thanks for the hint!
>
> Adrian
>
> -----Original Message-----
> From: Akhilesh Pathodia [mailto:pathodia.akhilesh@gmail.com]
> Sent: 07 April 2017 09:57
> To: users@kafka.apache.org
> Subject: Re: Topic deletion
>
> I am not sure but kafka delete command does not delete the topic
> actually, it only marks it for deletion. Probably it is fixed in later
> version of kafka.
>
> On Fri, Apr 7, 2017 at 2:14 PM, Adrian McCague
> <Ad...@zopa.com>
> wrote:
>
> > Hi Akhilesh,
> >
> > Why would this approach need to be taken over the kafka-topics tool,
> > out of interest?
> >
> > Thanks
> > Adrian
> >
> > -----Original Message-----
> > From: Akhilesh Pathodia [mailto:pathodia.akhilesh@gmail.com]
> > Sent: 07 April 2017 09:37
> > To: users@kafka.apache.org
> > Subject: Re: Topic deletion
> >
> > Hi Adrian,
> >
> > You will have to delete the broker directory from zookeeper. This
> > can be done from zookeeper cli. Connect to zookeeper cli using
> > below
> command:
> >
> > zookeeper-client -server <ZOOKEEPER_URI>
> >
> > Then run below command :
> >
> > rmr /brokers/topics/<TOPIC_NAME>
> >
> > Thanks,
> > AKhilesh
> >
> > On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague
> > <Ad...@zopa.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I am trying to understand topic deletion in kafka, there appears
> > > to be very little documentation or answers on how this works.
> > > Typically they just say to turn on the feature on the broker (in my case it is).
> > >
> > > I executed:
> > > Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
> > >
> > > Running this again yields:
> > > Topic mytopic is already marked for deletion.
> > >
> > > --describe yields:
> > > Topic: mytopic PartitionCount:6 ReplicationFactor:3
> Configs:
> > > retention.ms=0
> > > Topic: mytopic Partition: 0 Leader: -1 Replicas:
> > > 1006,1001,1005 Isr:
> > > Topic mytopic Partition: 1 Leader: -1 Replicas:
> > > 1001,1005,1003 Isr:
> > > Topic: mytopic Partition: 2 Leader: -1 Replicas:
> > > 1005,1003,1004 Isr:
> > > Topic: mytopic Partition: 3 Leader: -1 Replicas:
> > > 1003,1004,1007 Isr:
> > > Topic: mytopic Partition: 4 Leader: -1 Replicas:
> > > 1004,1007,1006 Isr:
> > > Topic: mytopic Partition: 5 Leader: -1 Replicas:
> > > 1007,1006,1001 Isr:
> > >
> > > You can see that the deletion mark has meant that the Leader is -1.
> > > Also I read somewhere that retention needs to be set to something
> > > low to trigger the deletion, hence the config of retention.ms=0
> > >
> > > Consumers (or streams in my case) no longer see the topic:
> > > org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
> > > topology building: stream-thread [StreamThread-1] Topic not found:
> > > mytopic
> > >
> > > And I can't create a new topic in its place:
> > > [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.
> > errors.TopicExistsException:
> > > Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
> > >
> > > I am a little lost as to where to go next, could someone explain
> > > how topic deletion is actually applied when a topic is 'marked'
> > > for deletion as that may help trigger it.
> > >
> > > Thanks!
> > > Adrian
> > >
> > >
> >
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Topic deletion
Posted by Mayuresh Gharat <gh...@gmail.com>.
Hi Adrian,
When you delete a topic, it is marked a under /admin/delete_topics path in
zookeeper. Once that is done, controller gets a notification that a topic
is marked for delete. The controller then notifies the replcias of the
topic-partitions to delete the data for the topic and the topic itself.
After KAFKA-1911, delete topic is made asynchronous and faster. The
replicas on receiving the notification from controller will rename the
topic-partition directory to something like topic-partition.delete and
return to controller saying delete topic is done. The controller when it
receives the ack from all the replicas that topic has been deleted, it will
delete the znode under /admin/delete_topics path. Your delete topic is
successful at this stage. The renamed directory gets removed asynchronously
on those replicas brokers at some later point in time.
Also remember, if you topic-partitions are been reassigned, delete topic
will not be triggered till the reassignment is finished.
We at Linkedin, have been suing Delete topic for a while now and it has
been working fine after KAFKA-1911.
Hope this helps.
Thanks,
Mayuresh
On Fri, Apr 7, 2017 at 2:15 AM, Adrian McCague <Ad...@zopa.com>
wrote:
> Indeed, I wish it was explained somewhere what marked for deletion does,
> and what process handles it. I will have to go and investigate the source.
>
> I can confirm the zkCli did the trick, thanks for the hint!
>
> Adrian
>
> -----Original Message-----
> From: Akhilesh Pathodia [mailto:pathodia.akhilesh@gmail.com]
> Sent: 07 April 2017 09:57
> To: users@kafka.apache.org
> Subject: Re: Topic deletion
>
> I am not sure but kafka delete command does not delete the topic actually,
> it only marks it for deletion. Probably it is fixed in later version of
> kafka.
>
> On Fri, Apr 7, 2017 at 2:14 PM, Adrian McCague <Ad...@zopa.com>
> wrote:
>
> > Hi Akhilesh,
> >
> > Why would this approach need to be taken over the kafka-topics tool,
> > out of interest?
> >
> > Thanks
> > Adrian
> >
> > -----Original Message-----
> > From: Akhilesh Pathodia [mailto:pathodia.akhilesh@gmail.com]
> > Sent: 07 April 2017 09:37
> > To: users@kafka.apache.org
> > Subject: Re: Topic deletion
> >
> > Hi Adrian,
> >
> > You will have to delete the broker directory from zookeeper. This can
> > be done from zookeeper cli. Connect to zookeeper cli using below
> command:
> >
> > zookeeper-client -server <ZOOKEEPER_URI>
> >
> > Then run below command :
> >
> > rmr /brokers/topics/<TOPIC_NAME>
> >
> > Thanks,
> > AKhilesh
> >
> > On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague
> > <Ad...@zopa.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I am trying to understand topic deletion in kafka, there appears to
> > > be very little documentation or answers on how this works. Typically
> > > they just say to turn on the feature on the broker (in my case it is).
> > >
> > > I executed:
> > > Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
> > >
> > > Running this again yields:
> > > Topic mytopic is already marked for deletion.
> > >
> > > --describe yields:
> > > Topic: mytopic PartitionCount:6 ReplicationFactor:3
> Configs:
> > > retention.ms=0
> > > Topic: mytopic Partition: 0 Leader: -1 Replicas:
> > > 1006,1001,1005 Isr:
> > > Topic mytopic Partition: 1 Leader: -1 Replicas:
> > > 1001,1005,1003 Isr:
> > > Topic: mytopic Partition: 2 Leader: -1 Replicas:
> > > 1005,1003,1004 Isr:
> > > Topic: mytopic Partition: 3 Leader: -1 Replicas:
> > > 1003,1004,1007 Isr:
> > > Topic: mytopic Partition: 4 Leader: -1 Replicas:
> > > 1004,1007,1006 Isr:
> > > Topic: mytopic Partition: 5 Leader: -1 Replicas:
> > > 1007,1006,1001 Isr:
> > >
> > > You can see that the deletion mark has meant that the Leader is -1.
> > > Also I read somewhere that retention needs to be set to something
> > > low to trigger the deletion, hence the config of retention.ms=0
> > >
> > > Consumers (or streams in my case) no longer see the topic:
> > > org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
> > > topology building: stream-thread [StreamThread-1] Topic not found:
> > > mytopic
> > >
> > > And I can't create a new topic in its place:
> > > [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.
> > errors.TopicExistsException:
> > > Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
> > >
> > > I am a little lost as to where to go next, could someone explain how
> > > topic deletion is actually applied when a topic is 'marked' for
> > > deletion as that may help trigger it.
> > >
> > > Thanks!
> > > Adrian
> > >
> > >
> >
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
RE: Topic deletion
Posted by Adrian McCague <Ad...@zopa.com>.
Indeed, I wish it was explained somewhere what marked for deletion does, and what process handles it. I will have to go and investigate the source.
I can confirm the zkCli did the trick, thanks for the hint!
Adrian
-----Original Message-----
From: Akhilesh Pathodia [mailto:pathodia.akhilesh@gmail.com]
Sent: 07 April 2017 09:57
To: users@kafka.apache.org
Subject: Re: Topic deletion
I am not sure but kafka delete command does not delete the topic actually, it only marks it for deletion. Probably it is fixed in later version of kafka.
On Fri, Apr 7, 2017 at 2:14 PM, Adrian McCague <Ad...@zopa.com>
wrote:
> Hi Akhilesh,
>
> Why would this approach need to be taken over the kafka-topics tool,
> out of interest?
>
> Thanks
> Adrian
>
> -----Original Message-----
> From: Akhilesh Pathodia [mailto:pathodia.akhilesh@gmail.com]
> Sent: 07 April 2017 09:37
> To: users@kafka.apache.org
> Subject: Re: Topic deletion
>
> Hi Adrian,
>
> You will have to delete the broker directory from zookeeper. This can
> be done from zookeeper cli. Connect to zookeeper cli using below command:
>
> zookeeper-client -server <ZOOKEEPER_URI>
>
> Then run below command :
>
> rmr /brokers/topics/<TOPIC_NAME>
>
> Thanks,
> AKhilesh
>
> On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague
> <Ad...@zopa.com>
> wrote:
>
> > Hi all,
> >
> > I am trying to understand topic deletion in kafka, there appears to
> > be very little documentation or answers on how this works. Typically
> > they just say to turn on the feature on the broker (in my case it is).
> >
> > I executed:
> > Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
> >
> > Running this again yields:
> > Topic mytopic is already marked for deletion.
> >
> > --describe yields:
> > Topic: mytopic PartitionCount:6 ReplicationFactor:3 Configs:
> > retention.ms=0
> > Topic: mytopic Partition: 0 Leader: -1 Replicas:
> > 1006,1001,1005 Isr:
> > Topic mytopic Partition: 1 Leader: -1 Replicas:
> > 1001,1005,1003 Isr:
> > Topic: mytopic Partition: 2 Leader: -1 Replicas:
> > 1005,1003,1004 Isr:
> > Topic: mytopic Partition: 3 Leader: -1 Replicas:
> > 1003,1004,1007 Isr:
> > Topic: mytopic Partition: 4 Leader: -1 Replicas:
> > 1004,1007,1006 Isr:
> > Topic: mytopic Partition: 5 Leader: -1 Replicas:
> > 1007,1006,1001 Isr:
> >
> > You can see that the deletion mark has meant that the Leader is -1.
> > Also I read somewhere that retention needs to be set to something
> > low to trigger the deletion, hence the config of retention.ms=0
> >
> > Consumers (or streams in my case) no longer see the topic:
> > org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
> > topology building: stream-thread [StreamThread-1] Topic not found:
> > mytopic
> >
> > And I can't create a new topic in its place:
> > [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.
> errors.TopicExistsException:
> > Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
> >
> > I am a little lost as to where to go next, could someone explain how
> > topic deletion is actually applied when a topic is 'marked' for
> > deletion as that may help trigger it.
> >
> > Thanks!
> > Adrian
> >
> >
>
Re: Topic deletion
Posted by Akhilesh Pathodia <pa...@gmail.com>.
I am not sure but kafka delete command does not delete the topic actually,
it only marks it for deletion. Probably it is fixed in later version of
kafka.
On Fri, Apr 7, 2017 at 2:14 PM, Adrian McCague <Ad...@zopa.com>
wrote:
> Hi Akhilesh,
>
> Why would this approach need to be taken over the kafka-topics tool, out
> of interest?
>
> Thanks
> Adrian
>
> -----Original Message-----
> From: Akhilesh Pathodia [mailto:pathodia.akhilesh@gmail.com]
> Sent: 07 April 2017 09:37
> To: users@kafka.apache.org
> Subject: Re: Topic deletion
>
> Hi Adrian,
>
> You will have to delete the broker directory from zookeeper. This can be
> done from zookeeper cli. Connect to zookeeper cli using below command:
>
> zookeeper-client -server <ZOOKEEPER_URI>
>
> Then run below command :
>
> rmr /brokers/topics/<TOPIC_NAME>
>
> Thanks,
> AKhilesh
>
> On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague <Ad...@zopa.com>
> wrote:
>
> > Hi all,
> >
> > I am trying to understand topic deletion in kafka, there appears to be
> > very little documentation or answers on how this works. Typically they
> > just say to turn on the feature on the broker (in my case it is).
> >
> > I executed:
> > Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
> >
> > Running this again yields:
> > Topic mytopic is already marked for deletion.
> >
> > --describe yields:
> > Topic: mytopic PartitionCount:6 ReplicationFactor:3 Configs:
> > retention.ms=0
> > Topic: mytopic Partition: 0 Leader: -1 Replicas:
> > 1006,1001,1005 Isr:
> > Topic mytopic Partition: 1 Leader: -1 Replicas:
> > 1001,1005,1003 Isr:
> > Topic: mytopic Partition: 2 Leader: -1 Replicas:
> > 1005,1003,1004 Isr:
> > Topic: mytopic Partition: 3 Leader: -1 Replicas:
> > 1003,1004,1007 Isr:
> > Topic: mytopic Partition: 4 Leader: -1 Replicas:
> > 1004,1007,1006 Isr:
> > Topic: mytopic Partition: 5 Leader: -1 Replicas:
> > 1007,1006,1001 Isr:
> >
> > You can see that the deletion mark has meant that the Leader is -1.
> > Also I read somewhere that retention needs to be set to something low
> > to trigger the deletion, hence the config of retention.ms=0
> >
> > Consumers (or streams in my case) no longer see the topic:
> > org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
> > topology building: stream-thread [StreamThread-1] Topic not found:
> > mytopic
> >
> > And I can't create a new topic in its place:
> > [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.
> errors.TopicExistsException:
> > Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
> >
> > I am a little lost as to where to go next, could someone explain how
> > topic deletion is actually applied when a topic is 'marked' for
> > deletion as that may help trigger it.
> >
> > Thanks!
> > Adrian
> >
> >
>
RE: Topic deletion
Posted by Adrian McCague <Ad...@zopa.com>.
Hi Akhilesh,
Why would this approach need to be taken over the kafka-topics tool, out of interest?
Thanks
Adrian
-----Original Message-----
From: Akhilesh Pathodia [mailto:pathodia.akhilesh@gmail.com]
Sent: 07 April 2017 09:37
To: users@kafka.apache.org
Subject: Re: Topic deletion
Hi Adrian,
You will have to delete the broker directory from zookeeper. This can be done from zookeeper cli. Connect to zookeeper cli using below command:
zookeeper-client -server <ZOOKEEPER_URI>
Then run below command :
rmr /brokers/topics/<TOPIC_NAME>
Thanks,
AKhilesh
On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague <Ad...@zopa.com>
wrote:
> Hi all,
>
> I am trying to understand topic deletion in kafka, there appears to be
> very little documentation or answers on how this works. Typically they
> just say to turn on the feature on the broker (in my case it is).
>
> I executed:
> Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
>
> Running this again yields:
> Topic mytopic is already marked for deletion.
>
> --describe yields:
> Topic: mytopic PartitionCount:6 ReplicationFactor:3 Configs:
> retention.ms=0
> Topic: mytopic Partition: 0 Leader: -1 Replicas:
> 1006,1001,1005 Isr:
> Topic mytopic Partition: 1 Leader: -1 Replicas:
> 1001,1005,1003 Isr:
> Topic: mytopic Partition: 2 Leader: -1 Replicas:
> 1005,1003,1004 Isr:
> Topic: mytopic Partition: 3 Leader: -1 Replicas:
> 1003,1004,1007 Isr:
> Topic: mytopic Partition: 4 Leader: -1 Replicas:
> 1004,1007,1006 Isr:
> Topic: mytopic Partition: 5 Leader: -1 Replicas:
> 1007,1006,1001 Isr:
>
> You can see that the deletion mark has meant that the Leader is -1.
> Also I read somewhere that retention needs to be set to something low
> to trigger the deletion, hence the config of retention.ms=0
>
> Consumers (or streams in my case) no longer see the topic:
> org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
> topology building: stream-thread [StreamThread-1] Topic not found:
> mytopic
>
> And I can't create a new topic in its place:
> [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.errors.TopicExistsException:
> Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
>
> I am a little lost as to where to go next, could someone explain how
> topic deletion is actually applied when a topic is 'marked' for
> deletion as that may help trigger it.
>
> Thanks!
> Adrian
>
>
Re: Topic deletion
Posted by Akhilesh Pathodia <pa...@gmail.com>.
Hi Adrian,
You will have to delete the broker directory from zookeeper. This can be
done from zookeeper cli. Connect to zookeeper cli using below command:
zookeeper-client -server <ZOOKEEPER_URI>
Then run below command :
rmr /brokers/topics/<TOPIC_NAME>
Thanks,
AKhilesh
On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague <Ad...@zopa.com>
wrote:
> Hi all,
>
> I am trying to understand topic deletion in kafka, there appears to be
> very little documentation or answers on how this works. Typically they just
> say to turn on the feature on the broker (in my case it is).
>
> I executed:
> Kafka-topics.bat -delete -zookeeper keeperhere -topic mytopic
>
> Running this again yields:
> Topic mytopic is already marked for deletion.
>
> --describe yields:
> Topic: mytopic PartitionCount:6 ReplicationFactor:3 Configs:
> retention.ms=0
> Topic: mytopic Partition: 0 Leader: -1 Replicas:
> 1006,1001,1005 Isr:
> Topic mytopic Partition: 1 Leader: -1 Replicas:
> 1001,1005,1003 Isr:
> Topic: mytopic Partition: 2 Leader: -1 Replicas:
> 1005,1003,1004 Isr:
> Topic: mytopic Partition: 3 Leader: -1 Replicas:
> 1003,1004,1007 Isr:
> Topic: mytopic Partition: 4 Leader: -1 Replicas:
> 1004,1007,1006 Isr:
> Topic: mytopic Partition: 5 Leader: -1 Replicas:
> 1007,1006,1001 Isr:
>
> You can see that the deletion mark has meant that the Leader is -1.
> Also I read somewhere that retention needs to be set to something low to
> trigger the deletion, hence the config of retention.ms=0
>
> Consumers (or streams in my case) no longer see the topic:
> org.apache.kafka.streams.errors.TopologyBuilderException: Invalid
> topology building: stream-thread [StreamThread-1] Topic not found: mytopic
>
> And I can't create a new topic in its place:
> [2017-04-06 18:26:00,702] ERROR org.apache.kafka.common.errors.TopicExistsException:
> Topic 'mytopic' already exists. (kafka.admin.TopicCommand$)
>
> I am a little lost as to where to go next, could someone explain how topic
> deletion is actually applied when a topic is 'marked' for deletion as that
> may help trigger it.
>
> Thanks!
> Adrian
>
>