You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Yu, Libo " <li...@citi.com> on 2013/08/22 19:30:47 UTC
delete a topic
Hi team,
When I delete a topic, the topic is deleted from zookeeper but its log files are not deleted from
Brokers.
When I restart a broker, the broker will try to sync the log files whose topic has been deleted.
Manually deleting the log files will resolve the issue. Should broker ignore log files whose topic
has been deleted?
Regards,
Libo
RE: delete a topic
Posted by "Yu, Libo " <li...@citi.com>.
Thanks, Neha.
Regards,
Libo
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Thursday, August 22, 2013 1:57 PM
To: users@kafka.apache.org
Subject: Re: delete a topic
We currently don't have the delete topic feature in Kafka 0.8. So any manual attempts to do so might have a negative impact on functionality.
Thanks,
Neha
On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com> wrote:
> Hi team,
>
> When I delete a topic, the topic is deleted from zookeeper but its log
> files are not deleted from Brokers.
>
> When I restart a broker, the broker will try to sync the log files
> whose topic has been deleted.
> Manually deleting the log files will resolve the issue. Should broker
> ignore log files whose topic has been deleted?
>
> Regards,
>
> Libo
>
>
Re: delete a topic
Posted by Andrew Otto <ot...@wikimedia.org>.
I maintain a kafka wrapper shell script wrapper for our Kafka packaging. I've noticed that 0.8 has several bin/*topic.sh scripts, while trunk has as single bin/kafka-topics.sh, which uses the TopicCommand class.
If you are discussing support for delete-topics in 0.8, perhaps the TopicCommand class and script that trunk has should be merged into 0.8 sooner rather than later? (I'm really only mentioning this because TopicCommand makes my wrapper script a bit simpler. :) )
-Andrew Otto
On Aug 23, 2013, at 5:28 PM, Jay Kreps <ja...@gmail.com> wrote:
> I mean I think the meta point Neha is making, is that clearly we need
> tooling support for topic deletion. We just haven't done it yet. This is
> kind of silly but works okay for us temporarily (we don't really delete
> topics).
>
> -Jay
>
>
> On Fri, Aug 23, 2013 at 2:22 PM, Yu, Libo <li...@citi.com> wrote:
>
>> Here is the list of paths that should be deleted:
>> /brokers/topics/my_topic
>> /consumers/[groupId]/owners/my_topic
>> /consumers/[groupId]/offsets/my_topic (if the topic has been consumed)
>> Let me know if I miss anything. Thanks.
>>
>> Regards,
>>
>> Libo
>>
>>
>> -----Original Message-----
>> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
>> Sent: Friday, August 23, 2013 4:35 PM
>> To: users@kafka.apache.org
>> Subject: Re: delete a topic
>>
>> You can either try using rmr on zookeeper client 3.4.x (./zkCli.sh rmr
>> /node) or write your own script that does a recursive delete -
>> https://issues.apache.org/jira/browse/ZOOKEEPER-729
>>
>> Thanks,
>> Neha
>>
>>
>> On Fri, Aug 23, 2013 at 10:52 AM, Yu, Libo <li...@citi.com> wrote:
>>
>>> I will give it a try. I know how to delete log files. But to delete
>>> the zookeeper data, do I only need to run the delete script?
>>>
>>> Regards,
>>>
>>> Libo
>>>
>>>
>>> -----Original Message-----
>>> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
>>> Sent: Friday, August 23, 2013 1:43 PM
>>> To: users@kafka.apache.org
>>> Subject: Re: delete a topic
>>>
>>> Without the proper functionality of delete topic, I'm not sure if
>>> deleting the zookeeper and kafka data just for that topic is enough or
>>> not. On the surface, this approach seems sufficient, but I'm not sure
>>> of all the consequences of doing that. Also, remember to bounce the
>>> entire cluster once you've deleted the zookeeper and kafka data for the
>> topic in question.
>>>
>>> Can you give it a try and let us know how it went?
>>>
>>> Thanks,
>>> Neha
>>>
>>>
>>> On Fri, Aug 23, 2013 at 10:15 AM, Yu, Libo <li...@citi.com> wrote:
>>>
>>>> Hi Neha,
>>>>
>>>> One more questions. Assume I want to delete a topic. When you say
>>>> deleting zookeeper data and kafka data, do you mean deleting
>>>> zookeeper data and kafka data for ALL the topics or only for that
>>>> particular topic?
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Libo
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
>>>> Sent: Friday, August 23, 2013 11:33 AM
>>>> To: users@kafka.apache.org
>>>> Subject: Re: delete a topic
>>>>
>>>> By that, I meant bringing down all brokers, deleting zookeeper data
>>>> and kafka data and restarting the brokers. I suspect attempting a
>>>> delete topic might have caused something bad to happen on the ISR
>>>> side. It will be great if you can start clean and then see if the
>>>> ISR is
>>> still an issue.
>>>>
>>>> Also, I filed a bug to include a tool that makes it easy to check
>>>> the replica lag and progress, while troubleshooting ISR membership
>>>> issues
>>>> -
>>>> https://issues.apache.org/jira/browse/KAFKA-1021
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>>
>>>> On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
>>>>
>>>>> Hi Neha,
>>>>>
>>>>> "Wipe out the cluster" Do you mean you uninstall the cluster and
>>>>> reinstall it?
>>>>> Or you just delete all kafka data and zookeeper data for the cluster?
>>>>> This is not a blocking issue for us. Our blocking issue is that
>>>>> some broker will fall out of ISR and never get back to it. We have
>>>>> observed that from more that one clusters. I am not sure if it is
>>>>> related to the deleting issue.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Libo
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
>>>>> Sent: Thursday, August 22, 2013 5:01 PM
>>>>> To: users@kafka.apache.org
>>>>> Subject: Re: delete a topic
>>>>>
>>>>> In production, we are not deleting topics yet. In test
>>>>> environments, if we have to delete topics, we wipe out the
>>>>> cluster. If this is a feature that most users are blocked on, I
>>>>> think it makes sense to prioritize
>> https://issues.apache.org/jira/browse/KAFKA-330.
>>>>>
>>>>> Thanks,
>>>>> Neha
>>>>>
>>>>>
>>>>> On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis
>>>>> <vk...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> How do you guys delete a topic if such need arise?
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
>>>>>> <neha.narkhede@gmail.com
>>>>>>> wrote:
>>>>>>
>>>>>>> I mentioned this in a different thread. I think the
>>>>>> 0.8.0-beta1-candidate1
>>>>>>> includes the script, but we have removed it later. I think
>>>>>>> when we
>>>>>> publish
>>>>>>> 0.8-final, this should be resolved.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Neha
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
>>>>>>> <neha.narkhede@gmail.com
>>>>>>>> wrote:
>>>>>>>
>>>>>>>> Vadim,
>>>>>>>>
>>>>>>>> The JIRA tracking this feature is
>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-330.
>>>>>>>> Until we have this feature, we should remove any scripts
>>>>>>>> that give the impression of deleting a topic, as Jay
>>>>>>>> suggested. I tried looking under bin/ but couldn't find any
>>>>>>>> delete topic or partition
>>>>> script.
>>>>>>>>
>>>>>>>> Can you tell us which script you are using?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Neha
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
>>>>>>>> <vkeylis2009@gmail.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Jay or Neha. What is the process of deleting the topic if
>>>>>>>>> the script included in the repository creates false
>> impression?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Vadim
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps
>>>>>>>>> <ja...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> We should really remove the delete script from 0.8 if we
>>>>>>>>>> plan to
>>>>>>> release
>>>>>>>>>> without delete support. Right now it appears to work but
>>>>>>>>>> doesn't
>>>>>> which
>>>>>>>>> is
>>>>>>>>>> clearly not good.
>>>>>>>>>>
>>>>>>>>>> -Jay
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
>>>>>>>>> neha.narkhede@gmail.com
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> We currently don't have the delete topic feature in
>>>>>>>>>>> Kafka 0.8. So
>>>>>>> any
>>>>>>>>>>> manual attempts to do so might have a negative impact
>>>>>>>>>>> on
>>>>>>>>> functionality.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Neha
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo
>>>>>>>>>>> <li...@citi.com>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi team,
>>>>>>>>>>>>
>>>>>>>>>>>> When I delete a topic, the topic is deleted from
>>>>>>>>>>>> zookeeper but
>>>>>> its
>>>>>>>>> log
>>>>>>>>>>>> files are not deleted from Brokers.
>>>>>>>>>>>>
>>>>>>>>>>>> When I restart a broker, the broker will try to sync
>>>>>>>>>>>> the log
>>>>>>> files
>>>>>>>>>> whose
>>>>>>>>>>>> topic has been deleted.
>>>>>>>>>>>> Manually deleting the log files will resolve the issue.
>>>>>>>>>>>> Should
>>>>>>>>> broker
>>>>>>>>>>>> ignore log files whose topic has been deleted?
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Libo
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: delete a topic
Posted by Jay Kreps <ja...@gmail.com>.
I mean I think the meta point Neha is making, is that clearly we need
tooling support for topic deletion. We just haven't done it yet. This is
kind of silly but works okay for us temporarily (we don't really delete
topics).
-Jay
On Fri, Aug 23, 2013 at 2:22 PM, Yu, Libo <li...@citi.com> wrote:
> Here is the list of paths that should be deleted:
> /brokers/topics/my_topic
> /consumers/[groupId]/owners/my_topic
> /consumers/[groupId]/offsets/my_topic (if the topic has been consumed)
> Let me know if I miss anything. Thanks.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Friday, August 23, 2013 4:35 PM
> To: users@kafka.apache.org
> Subject: Re: delete a topic
>
> You can either try using rmr on zookeeper client 3.4.x (./zkCli.sh rmr
> /node) or write your own script that does a recursive delete -
> https://issues.apache.org/jira/browse/ZOOKEEPER-729
>
> Thanks,
> Neha
>
>
> On Fri, Aug 23, 2013 at 10:52 AM, Yu, Libo <li...@citi.com> wrote:
>
> > I will give it a try. I know how to delete log files. But to delete
> > the zookeeper data, do I only need to run the delete script?
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Friday, August 23, 2013 1:43 PM
> > To: users@kafka.apache.org
> > Subject: Re: delete a topic
> >
> > Without the proper functionality of delete topic, I'm not sure if
> > deleting the zookeeper and kafka data just for that topic is enough or
> > not. On the surface, this approach seems sufficient, but I'm not sure
> > of all the consequences of doing that. Also, remember to bounce the
> > entire cluster once you've deleted the zookeeper and kafka data for the
> topic in question.
> >
> > Can you give it a try and let us know how it went?
> >
> > Thanks,
> > Neha
> >
> >
> > On Fri, Aug 23, 2013 at 10:15 AM, Yu, Libo <li...@citi.com> wrote:
> >
> > > Hi Neha,
> > >
> > > One more questions. Assume I want to delete a topic. When you say
> > > deleting zookeeper data and kafka data, do you mean deleting
> > > zookeeper data and kafka data for ALL the topics or only for that
> > > particular topic?
> > >
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > Sent: Friday, August 23, 2013 11:33 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: delete a topic
> > >
> > > By that, I meant bringing down all brokers, deleting zookeeper data
> > > and kafka data and restarting the brokers. I suspect attempting a
> > > delete topic might have caused something bad to happen on the ISR
> > > side. It will be great if you can start clean and then see if the
> > > ISR is
> > still an issue.
> > >
> > > Also, I filed a bug to include a tool that makes it easy to check
> > > the replica lag and progress, while troubleshooting ISR membership
> > > issues
> > > -
> > > https://issues.apache.org/jira/browse/KAFKA-1021
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
> > >
> > > > Hi Neha,
> > > >
> > > > "Wipe out the cluster" Do you mean you uninstall the cluster and
> > > > reinstall it?
> > > > Or you just delete all kafka data and zookeeper data for the cluster?
> > > > This is not a blocking issue for us. Our blocking issue is that
> > > > some broker will fall out of ISR and never get back to it. We have
> > > > observed that from more that one clusters. I am not sure if it is
> > > > related to the deleting issue.
> > > >
> > > > Regards,
> > > >
> > > > Libo
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > > Sent: Thursday, August 22, 2013 5:01 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: delete a topic
> > > >
> > > > In production, we are not deleting topics yet. In test
> > > > environments, if we have to delete topics, we wipe out the
> > > > cluster. If this is a feature that most users are blocked on, I
> > > > think it makes sense to prioritize
> https://issues.apache.org/jira/browse/KAFKA-330.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis
> > > > <vk...@gmail.com>
> > > > wrote:
> > > >
> > > > > How do you guys delete a topic if such need arise?
> > > > >
> > > > >
> > > > > On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
> > > > > <neha.narkhede@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > I mentioned this in a different thread. I think the
> > > > > 0.8.0-beta1-candidate1
> > > > > > includes the script, but we have removed it later. I think
> > > > > > when we
> > > > > publish
> > > > > > 0.8-final, this should be resolved.
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
> > > > > > <neha.narkhede@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Vadim,
> > > > > > >
> > > > > > > The JIRA tracking this feature is
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-330.
> > > > > > > Until we have this feature, we should remove any scripts
> > > > > > > that give the impression of deleting a topic, as Jay
> > > > > > > suggested. I tried looking under bin/ but couldn't find any
> > > > > > > delete topic or partition
> > > > script.
> > > > > > >
> > > > > > > Can you tell us which script you are using?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Neha
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
> > > > > > ><vkeylis2009@gmail.com
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Jay or Neha. What is the process of deleting the topic if
> > > > > > >> the script included in the repository creates false
> impression?
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Vadim
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps
> > > > > > >> <ja...@gmail.com>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > We should really remove the delete script from 0.8 if we
> > > > > > >> > plan to
> > > > > > release
> > > > > > >> > without delete support. Right now it appears to work but
> > > > > > >> > doesn't
> > > > > which
> > > > > > >> is
> > > > > > >> > clearly not good.
> > > > > > >> >
> > > > > > >> > -Jay
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> > > > > > >> neha.narkhede@gmail.com
> > > > > > >> > >wrote:
> > > > > > >> >
> > > > > > >> > > We currently don't have the delete topic feature in
> > > > > > >> > > Kafka 0.8. So
> > > > > > any
> > > > > > >> > > manual attempts to do so might have a negative impact
> > > > > > >> > > on
> > > > > > >> functionality.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Neha
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo
> > > > > > >> > > <li...@citi.com>
> > > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi team,
> > > > > > >> > > >
> > > > > > >> > > > When I delete a topic, the topic is deleted from
> > > > > > >> > > > zookeeper but
> > > > > its
> > > > > > >> log
> > > > > > >> > > > files are not deleted from Brokers.
> > > > > > >> > > >
> > > > > > >> > > > When I restart a broker, the broker will try to sync
> > > > > > >> > > > the log
> > > > > > files
> > > > > > >> > whose
> > > > > > >> > > > topic has been deleted.
> > > > > > >> > > > Manually deleting the log files will resolve the issue.
> > > > > > >> > > > Should
> > > > > > >> broker
> > > > > > >> > > > ignore log files whose topic has been deleted?
> > > > > > >> > > >
> > > > > > >> > > > Regards,
> > > > > > >> > > >
> > > > > > >> > > > Libo
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
RE: delete a topic
Posted by "Yu, Libo " <li...@citi.com>.
Here is the list of paths that should be deleted:
/brokers/topics/my_topic
/consumers/[groupId]/owners/my_topic
/consumers/[groupId]/offsets/my_topic (if the topic has been consumed)
Let me know if I miss anything. Thanks.
Regards,
Libo
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Friday, August 23, 2013 4:35 PM
To: users@kafka.apache.org
Subject: Re: delete a topic
You can either try using rmr on zookeeper client 3.4.x (./zkCli.sh rmr
/node) or write your own script that does a recursive delete -
https://issues.apache.org/jira/browse/ZOOKEEPER-729
Thanks,
Neha
On Fri, Aug 23, 2013 at 10:52 AM, Yu, Libo <li...@citi.com> wrote:
> I will give it a try. I know how to delete log files. But to delete
> the zookeeper data, do I only need to run the delete script?
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Friday, August 23, 2013 1:43 PM
> To: users@kafka.apache.org
> Subject: Re: delete a topic
>
> Without the proper functionality of delete topic, I'm not sure if
> deleting the zookeeper and kafka data just for that topic is enough or
> not. On the surface, this approach seems sufficient, but I'm not sure
> of all the consequences of doing that. Also, remember to bounce the
> entire cluster once you've deleted the zookeeper and kafka data for the topic in question.
>
> Can you give it a try and let us know how it went?
>
> Thanks,
> Neha
>
>
> On Fri, Aug 23, 2013 at 10:15 AM, Yu, Libo <li...@citi.com> wrote:
>
> > Hi Neha,
> >
> > One more questions. Assume I want to delete a topic. When you say
> > deleting zookeeper data and kafka data, do you mean deleting
> > zookeeper data and kafka data for ALL the topics or only for that
> > particular topic?
> >
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Friday, August 23, 2013 11:33 AM
> > To: users@kafka.apache.org
> > Subject: Re: delete a topic
> >
> > By that, I meant bringing down all brokers, deleting zookeeper data
> > and kafka data and restarting the brokers. I suspect attempting a
> > delete topic might have caused something bad to happen on the ISR
> > side. It will be great if you can start clean and then see if the
> > ISR is
> still an issue.
> >
> > Also, I filed a bug to include a tool that makes it easy to check
> > the replica lag and progress, while troubleshooting ISR membership
> > issues
> > -
> > https://issues.apache.org/jira/browse/KAFKA-1021
> >
> > Thanks,
> > Neha
> >
> >
> > On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
> >
> > > Hi Neha,
> > >
> > > "Wipe out the cluster" Do you mean you uninstall the cluster and
> > > reinstall it?
> > > Or you just delete all kafka data and zookeeper data for the cluster?
> > > This is not a blocking issue for us. Our blocking issue is that
> > > some broker will fall out of ISR and never get back to it. We have
> > > observed that from more that one clusters. I am not sure if it is
> > > related to the deleting issue.
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > Sent: Thursday, August 22, 2013 5:01 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: delete a topic
> > >
> > > In production, we are not deleting topics yet. In test
> > > environments, if we have to delete topics, we wipe out the
> > > cluster. If this is a feature that most users are blocked on, I
> > > think it makes sense to prioritize https://issues.apache.org/jira/browse/KAFKA-330.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis
> > > <vk...@gmail.com>
> > > wrote:
> > >
> > > > How do you guys delete a topic if such need arise?
> > > >
> > > >
> > > > On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
> > > > <neha.narkhede@gmail.com
> > > > >wrote:
> > > >
> > > > > I mentioned this in a different thread. I think the
> > > > 0.8.0-beta1-candidate1
> > > > > includes the script, but we have removed it later. I think
> > > > > when we
> > > > publish
> > > > > 0.8-final, this should be resolved.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
> > > > > <neha.narkhede@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Vadim,
> > > > > >
> > > > > > The JIRA tracking this feature is
> > > > > > https://issues.apache.org/jira/browse/KAFKA-330.
> > > > > > Until we have this feature, we should remove any scripts
> > > > > > that give the impression of deleting a topic, as Jay
> > > > > > suggested. I tried looking under bin/ but couldn't find any
> > > > > > delete topic or partition
> > > script.
> > > > > >
> > > > > > Can you tell us which script you are using?
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
> > > > > ><vkeylis2009@gmail.com
> > > > > >wrote:
> > > > > >
> > > > > >> Jay or Neha. What is the process of deleting the topic if
> > > > > >> the script included in the repository creates false impression?
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Vadim
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps
> > > > > >> <ja...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> > We should really remove the delete script from 0.8 if we
> > > > > >> > plan to
> > > > > release
> > > > > >> > without delete support. Right now it appears to work but
> > > > > >> > doesn't
> > > > which
> > > > > >> is
> > > > > >> > clearly not good.
> > > > > >> >
> > > > > >> > -Jay
> > > > > >> >
> > > > > >> >
> > > > > >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> > > > > >> neha.narkhede@gmail.com
> > > > > >> > >wrote:
> > > > > >> >
> > > > > >> > > We currently don't have the delete topic feature in
> > > > > >> > > Kafka 0.8. So
> > > > > any
> > > > > >> > > manual attempts to do so might have a negative impact
> > > > > >> > > on
> > > > > >> functionality.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Neha
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo
> > > > > >> > > <li...@citi.com>
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi team,
> > > > > >> > > >
> > > > > >> > > > When I delete a topic, the topic is deleted from
> > > > > >> > > > zookeeper but
> > > > its
> > > > > >> log
> > > > > >> > > > files are not deleted from Brokers.
> > > > > >> > > >
> > > > > >> > > > When I restart a broker, the broker will try to sync
> > > > > >> > > > the log
> > > > > files
> > > > > >> > whose
> > > > > >> > > > topic has been deleted.
> > > > > >> > > > Manually deleting the log files will resolve the issue.
> > > > > >> > > > Should
> > > > > >> broker
> > > > > >> > > > ignore log files whose topic has been deleted?
> > > > > >> > > >
> > > > > >> > > > Regards,
> > > > > >> > > >
> > > > > >> > > > Libo
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: delete a topic
Posted by Neha Narkhede <ne...@gmail.com>.
You can either try using rmr on zookeeper client 3.4.x (./zkCli.sh rmr
/node) or write your own script that does a recursive delete -
https://issues.apache.org/jira/browse/ZOOKEEPER-729
Thanks,
Neha
On Fri, Aug 23, 2013 at 10:52 AM, Yu, Libo <li...@citi.com> wrote:
> I will give it a try. I know how to delete log files. But to delete the
> zookeeper data,
> do I only need to run the delete script?
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Friday, August 23, 2013 1:43 PM
> To: users@kafka.apache.org
> Subject: Re: delete a topic
>
> Without the proper functionality of delete topic, I'm not sure if deleting
> the zookeeper and kafka data just for that topic is enough or not. On the
> surface, this approach seems sufficient, but I'm not sure of all the
> consequences of doing that. Also, remember to bounce the entire cluster
> once you've deleted the zookeeper and kafka data for the topic in question.
>
> Can you give it a try and let us know how it went?
>
> Thanks,
> Neha
>
>
> On Fri, Aug 23, 2013 at 10:15 AM, Yu, Libo <li...@citi.com> wrote:
>
> > Hi Neha,
> >
> > One more questions. Assume I want to delete a topic. When you say
> > deleting zookeeper data and kafka data, do you mean deleting zookeeper
> > data and kafka data for ALL the topics or only for that particular
> > topic?
> >
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Friday, August 23, 2013 11:33 AM
> > To: users@kafka.apache.org
> > Subject: Re: delete a topic
> >
> > By that, I meant bringing down all brokers, deleting zookeeper data
> > and kafka data and restarting the brokers. I suspect attempting a
> > delete topic might have caused something bad to happen on the ISR
> > side. It will be great if you can start clean and then see if the ISR is
> still an issue.
> >
> > Also, I filed a bug to include a tool that makes it easy to check the
> > replica lag and progress, while troubleshooting ISR membership issues
> > -
> > https://issues.apache.org/jira/browse/KAFKA-1021
> >
> > Thanks,
> > Neha
> >
> >
> > On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
> >
> > > Hi Neha,
> > >
> > > "Wipe out the cluster" Do you mean you uninstall the cluster and
> > > reinstall it?
> > > Or you just delete all kafka data and zookeeper data for the cluster?
> > > This is not a blocking issue for us. Our blocking issue is that some
> > > broker will fall out of ISR and never get back to it. We have
> > > observed that from more that one clusters. I am not sure if it is
> > > related to the deleting issue.
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> > > -----Original Message-----
> > > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > > Sent: Thursday, August 22, 2013 5:01 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: delete a topic
> > >
> > > In production, we are not deleting topics yet. In test environments,
> > > if we have to delete topics, we wipe out the cluster. If this is a
> > > feature that most users are blocked on, I think it makes sense to
> > > prioritize https://issues.apache.org/jira/browse/KAFKA-330.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis
> > > <vk...@gmail.com>
> > > wrote:
> > >
> > > > How do you guys delete a topic if such need arise?
> > > >
> > > >
> > > > On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
> > > > <neha.narkhede@gmail.com
> > > > >wrote:
> > > >
> > > > > I mentioned this in a different thread. I think the
> > > > 0.8.0-beta1-candidate1
> > > > > includes the script, but we have removed it later. I think when
> > > > > we
> > > > publish
> > > > > 0.8-final, this should be resolved.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
> > > > > <neha.narkhede@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Vadim,
> > > > > >
> > > > > > The JIRA tracking this feature is
> > > > > > https://issues.apache.org/jira/browse/KAFKA-330.
> > > > > > Until we have this feature, we should remove any scripts that
> > > > > > give the impression of deleting a topic, as Jay suggested. I
> > > > > > tried looking under bin/ but couldn't find any delete topic or
> > > > > > partition
> > > script.
> > > > > >
> > > > > > Can you tell us which script you are using?
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
> > > > > ><vkeylis2009@gmail.com
> > > > > >wrote:
> > > > > >
> > > > > >> Jay or Neha. What is the process of deleting the topic if the
> > > > > >> script included in the repository creates false impression?
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Vadim
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps
> > > > > >> <ja...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> > We should really remove the delete script from 0.8 if we
> > > > > >> > plan to
> > > > > release
> > > > > >> > without delete support. Right now it appears to work but
> > > > > >> > doesn't
> > > > which
> > > > > >> is
> > > > > >> > clearly not good.
> > > > > >> >
> > > > > >> > -Jay
> > > > > >> >
> > > > > >> >
> > > > > >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> > > > > >> neha.narkhede@gmail.com
> > > > > >> > >wrote:
> > > > > >> >
> > > > > >> > > We currently don't have the delete topic feature in Kafka
> > > > > >> > > 0.8. So
> > > > > any
> > > > > >> > > manual attempts to do so might have a negative impact on
> > > > > >> functionality.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Neha
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo
> > > > > >> > > <li...@citi.com>
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi team,
> > > > > >> > > >
> > > > > >> > > > When I delete a topic, the topic is deleted from
> > > > > >> > > > zookeeper but
> > > > its
> > > > > >> log
> > > > > >> > > > files are not deleted from Brokers.
> > > > > >> > > >
> > > > > >> > > > When I restart a broker, the broker will try to sync
> > > > > >> > > > the log
> > > > > files
> > > > > >> > whose
> > > > > >> > > > topic has been deleted.
> > > > > >> > > > Manually deleting the log files will resolve the issue.
> > > > > >> > > > Should
> > > > > >> broker
> > > > > >> > > > ignore log files whose topic has been deleted?
> > > > > >> > > >
> > > > > >> > > > Regards,
> > > > > >> > > >
> > > > > >> > > > Libo
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
RE: delete a topic
Posted by "Yu, Libo " <li...@citi.com>.
I will give it a try. I know how to delete log files. But to delete the zookeeper data,
do I only need to run the delete script?
Regards,
Libo
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Friday, August 23, 2013 1:43 PM
To: users@kafka.apache.org
Subject: Re: delete a topic
Without the proper functionality of delete topic, I'm not sure if deleting the zookeeper and kafka data just for that topic is enough or not. On the surface, this approach seems sufficient, but I'm not sure of all the consequences of doing that. Also, remember to bounce the entire cluster once you've deleted the zookeeper and kafka data for the topic in question.
Can you give it a try and let us know how it went?
Thanks,
Neha
On Fri, Aug 23, 2013 at 10:15 AM, Yu, Libo <li...@citi.com> wrote:
> Hi Neha,
>
> One more questions. Assume I want to delete a topic. When you say
> deleting zookeeper data and kafka data, do you mean deleting zookeeper
> data and kafka data for ALL the topics or only for that particular
> topic?
>
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Friday, August 23, 2013 11:33 AM
> To: users@kafka.apache.org
> Subject: Re: delete a topic
>
> By that, I meant bringing down all brokers, deleting zookeeper data
> and kafka data and restarting the brokers. I suspect attempting a
> delete topic might have caused something bad to happen on the ISR
> side. It will be great if you can start clean and then see if the ISR is still an issue.
>
> Also, I filed a bug to include a tool that makes it easy to check the
> replica lag and progress, while troubleshooting ISR membership issues
> -
> https://issues.apache.org/jira/browse/KAFKA-1021
>
> Thanks,
> Neha
>
>
> On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
>
> > Hi Neha,
> >
> > "Wipe out the cluster" Do you mean you uninstall the cluster and
> > reinstall it?
> > Or you just delete all kafka data and zookeeper data for the cluster?
> > This is not a blocking issue for us. Our blocking issue is that some
> > broker will fall out of ISR and never get back to it. We have
> > observed that from more that one clusters. I am not sure if it is
> > related to the deleting issue.
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Thursday, August 22, 2013 5:01 PM
> > To: users@kafka.apache.org
> > Subject: Re: delete a topic
> >
> > In production, we are not deleting topics yet. In test environments,
> > if we have to delete topics, we wipe out the cluster. If this is a
> > feature that most users are blocked on, I think it makes sense to
> > prioritize https://issues.apache.org/jira/browse/KAFKA-330.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis
> > <vk...@gmail.com>
> > wrote:
> >
> > > How do you guys delete a topic if such need arise?
> > >
> > >
> > > On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
> > > <neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > I mentioned this in a different thread. I think the
> > > 0.8.0-beta1-candidate1
> > > > includes the script, but we have removed it later. I think when
> > > > we
> > > publish
> > > > 0.8-final, this should be resolved.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
> > > > <neha.narkhede@gmail.com
> > > > >wrote:
> > > >
> > > > > Vadim,
> > > > >
> > > > > The JIRA tracking this feature is
> > > > > https://issues.apache.org/jira/browse/KAFKA-330.
> > > > > Until we have this feature, we should remove any scripts that
> > > > > give the impression of deleting a topic, as Jay suggested. I
> > > > > tried looking under bin/ but couldn't find any delete topic or
> > > > > partition
> > script.
> > > > >
> > > > > Can you tell us which script you are using?
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
> > > > ><vkeylis2009@gmail.com
> > > > >wrote:
> > > > >
> > > > >> Jay or Neha. What is the process of deleting the topic if the
> > > > >> script included in the repository creates false impression?
> > > > >>
> > > > >> Thanks,
> > > > >> Vadim
> > > > >>
> > > > >>
> > > > >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps
> > > > >> <ja...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> > We should really remove the delete script from 0.8 if we
> > > > >> > plan to
> > > > release
> > > > >> > without delete support. Right now it appears to work but
> > > > >> > doesn't
> > > which
> > > > >> is
> > > > >> > clearly not good.
> > > > >> >
> > > > >> > -Jay
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> > > > >> neha.narkhede@gmail.com
> > > > >> > >wrote:
> > > > >> >
> > > > >> > > We currently don't have the delete topic feature in Kafka
> > > > >> > > 0.8. So
> > > > any
> > > > >> > > manual attempts to do so might have a negative impact on
> > > > >> functionality.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Neha
> > > > >> > >
> > > > >> > >
> > > > >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo
> > > > >> > > <li...@citi.com>
> > > > wrote:
> > > > >> > >
> > > > >> > > > Hi team,
> > > > >> > > >
> > > > >> > > > When I delete a topic, the topic is deleted from
> > > > >> > > > zookeeper but
> > > its
> > > > >> log
> > > > >> > > > files are not deleted from Brokers.
> > > > >> > > >
> > > > >> > > > When I restart a broker, the broker will try to sync
> > > > >> > > > the log
> > > > files
> > > > >> > whose
> > > > >> > > > topic has been deleted.
> > > > >> > > > Manually deleting the log files will resolve the issue.
> > > > >> > > > Should
> > > > >> broker
> > > > >> > > > ignore log files whose topic has been deleted?
> > > > >> > > >
> > > > >> > > > Regards,
> > > > >> > > >
> > > > >> > > > Libo
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: delete a topic
Posted by Neha Narkhede <ne...@gmail.com>.
Without the proper functionality of delete topic, I'm not sure if deleting
the zookeeper and kafka data just for that topic is enough or not. On the
surface, this approach seems sufficient, but I'm not sure of all the
consequences of doing that. Also, remember to bounce the entire cluster
once you've deleted the zookeeper and kafka data for the topic in question.
Can you give it a try and let us know how it went?
Thanks,
Neha
On Fri, Aug 23, 2013 at 10:15 AM, Yu, Libo <li...@citi.com> wrote:
> Hi Neha,
>
> One more questions. Assume I want to delete a topic. When you say deleting
> zookeeper data and kafka data,
> do you mean deleting zookeeper data and kafka data for ALL the topics or
> only for that particular topic?
>
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Friday, August 23, 2013 11:33 AM
> To: users@kafka.apache.org
> Subject: Re: delete a topic
>
> By that, I meant bringing down all brokers, deleting zookeeper data and
> kafka data and restarting the brokers. I suspect attempting a delete topic
> might have caused something bad to happen on the ISR side. It will be great
> if you can start clean and then see if the ISR is still an issue.
>
> Also, I filed a bug to include a tool that makes it easy to check the
> replica lag and progress, while troubleshooting ISR membership issues -
> https://issues.apache.org/jira/browse/KAFKA-1021
>
> Thanks,
> Neha
>
>
> On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
>
> > Hi Neha,
> >
> > "Wipe out the cluster" Do you mean you uninstall the cluster and
> > reinstall it?
> > Or you just delete all kafka data and zookeeper data for the cluster?
> > This is not a blocking issue for us. Our blocking issue is that some
> > broker will fall out of ISR and never get back to it. We have observed
> > that from more that one clusters. I am not sure if it is related to
> > the deleting issue.
> >
> > Regards,
> >
> > Libo
> >
> >
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: Thursday, August 22, 2013 5:01 PM
> > To: users@kafka.apache.org
> > Subject: Re: delete a topic
> >
> > In production, we are not deleting topics yet. In test environments,
> > if we have to delete topics, we wipe out the cluster. If this is a
> > feature that most users are blocked on, I think it makes sense to
> > prioritize https://issues.apache.org/jira/browse/KAFKA-330.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis <vk...@gmail.com>
> > wrote:
> >
> > > How do you guys delete a topic if such need arise?
> > >
> > >
> > > On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
> > > <neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > I mentioned this in a different thread. I think the
> > > 0.8.0-beta1-candidate1
> > > > includes the script, but we have removed it later. I think when we
> > > publish
> > > > 0.8-final, this should be resolved.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
> > > > <neha.narkhede@gmail.com
> > > > >wrote:
> > > >
> > > > > Vadim,
> > > > >
> > > > > The JIRA tracking this feature is
> > > > > https://issues.apache.org/jira/browse/KAFKA-330.
> > > > > Until we have this feature, we should remove any scripts that
> > > > > give the impression of deleting a topic, as Jay suggested. I
> > > > > tried looking under bin/ but couldn't find any delete topic or
> > > > > partition
> > script.
> > > > >
> > > > > Can you tell us which script you are using?
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
> > > > ><vkeylis2009@gmail.com
> > > > >wrote:
> > > > >
> > > > >> Jay or Neha. What is the process of deleting the topic if the
> > > > >> script included in the repository creates false impression?
> > > > >>
> > > > >> Thanks,
> > > > >> Vadim
> > > > >>
> > > > >>
> > > > >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps
> > > > >> <ja...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> > We should really remove the delete script from 0.8 if we plan
> > > > >> > to
> > > > release
> > > > >> > without delete support. Right now it appears to work but
> > > > >> > doesn't
> > > which
> > > > >> is
> > > > >> > clearly not good.
> > > > >> >
> > > > >> > -Jay
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> > > > >> neha.narkhede@gmail.com
> > > > >> > >wrote:
> > > > >> >
> > > > >> > > We currently don't have the delete topic feature in Kafka
> > > > >> > > 0.8. So
> > > > any
> > > > >> > > manual attempts to do so might have a negative impact on
> > > > >> functionality.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Neha
> > > > >> > >
> > > > >> > >
> > > > >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo
> > > > >> > > <li...@citi.com>
> > > > wrote:
> > > > >> > >
> > > > >> > > > Hi team,
> > > > >> > > >
> > > > >> > > > When I delete a topic, the topic is deleted from
> > > > >> > > > zookeeper but
> > > its
> > > > >> log
> > > > >> > > > files are not deleted from Brokers.
> > > > >> > > >
> > > > >> > > > When I restart a broker, the broker will try to sync the
> > > > >> > > > log
> > > > files
> > > > >> > whose
> > > > >> > > > topic has been deleted.
> > > > >> > > > Manually deleting the log files will resolve the issue.
> > > > >> > > > Should
> > > > >> broker
> > > > >> > > > ignore log files whose topic has been deleted?
> > > > >> > > >
> > > > >> > > > Regards,
> > > > >> > > >
> > > > >> > > > Libo
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
RE: delete a topic
Posted by "Yu, Libo " <li...@citi.com>.
Hi Neha,
One more questions. Assume I want to delete a topic. When you say deleting zookeeper data and kafka data,
do you mean deleting zookeeper data and kafka data for ALL the topics or only for that particular topic?
Regards,
Libo
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Friday, August 23, 2013 11:33 AM
To: users@kafka.apache.org
Subject: Re: delete a topic
By that, I meant bringing down all brokers, deleting zookeeper data and kafka data and restarting the brokers. I suspect attempting a delete topic might have caused something bad to happen on the ISR side. It will be great if you can start clean and then see if the ISR is still an issue.
Also, I filed a bug to include a tool that makes it easy to check the replica lag and progress, while troubleshooting ISR membership issues -
https://issues.apache.org/jira/browse/KAFKA-1021
Thanks,
Neha
On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
> Hi Neha,
>
> "Wipe out the cluster" Do you mean you uninstall the cluster and
> reinstall it?
> Or you just delete all kafka data and zookeeper data for the cluster?
> This is not a blocking issue for us. Our blocking issue is that some
> broker will fall out of ISR and never get back to it. We have observed
> that from more that one clusters. I am not sure if it is related to
> the deleting issue.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Thursday, August 22, 2013 5:01 PM
> To: users@kafka.apache.org
> Subject: Re: delete a topic
>
> In production, we are not deleting topics yet. In test environments,
> if we have to delete topics, we wipe out the cluster. If this is a
> feature that most users are blocked on, I think it makes sense to
> prioritize https://issues.apache.org/jira/browse/KAFKA-330.
>
> Thanks,
> Neha
>
>
> On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis <vk...@gmail.com>
> wrote:
>
> > How do you guys delete a topic if such need arise?
> >
> >
> > On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
> > <neha.narkhede@gmail.com
> > >wrote:
> >
> > > I mentioned this in a different thread. I think the
> > 0.8.0-beta1-candidate1
> > > includes the script, but we have removed it later. I think when we
> > publish
> > > 0.8-final, this should be resolved.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
> > > <neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Vadim,
> > > >
> > > > The JIRA tracking this feature is
> > > > https://issues.apache.org/jira/browse/KAFKA-330.
> > > > Until we have this feature, we should remove any scripts that
> > > > give the impression of deleting a topic, as Jay suggested. I
> > > > tried looking under bin/ but couldn't find any delete topic or
> > > > partition
> script.
> > > >
> > > > Can you tell us which script you are using?
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
> > > ><vkeylis2009@gmail.com
> > > >wrote:
> > > >
> > > >> Jay or Neha. What is the process of deleting the topic if the
> > > >> script included in the repository creates false impression?
> > > >>
> > > >> Thanks,
> > > >> Vadim
> > > >>
> > > >>
> > > >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps
> > > >> <ja...@gmail.com>
> > > wrote:
> > > >>
> > > >> > We should really remove the delete script from 0.8 if we plan
> > > >> > to
> > > release
> > > >> > without delete support. Right now it appears to work but
> > > >> > doesn't
> > which
> > > >> is
> > > >> > clearly not good.
> > > >> >
> > > >> > -Jay
> > > >> >
> > > >> >
> > > >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> > > >> neha.narkhede@gmail.com
> > > >> > >wrote:
> > > >> >
> > > >> > > We currently don't have the delete topic feature in Kafka
> > > >> > > 0.8. So
> > > any
> > > >> > > manual attempts to do so might have a negative impact on
> > > >> functionality.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Neha
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo
> > > >> > > <li...@citi.com>
> > > wrote:
> > > >> > >
> > > >> > > > Hi team,
> > > >> > > >
> > > >> > > > When I delete a topic, the topic is deleted from
> > > >> > > > zookeeper but
> > its
> > > >> log
> > > >> > > > files are not deleted from Brokers.
> > > >> > > >
> > > >> > > > When I restart a broker, the broker will try to sync the
> > > >> > > > log
> > > files
> > > >> > whose
> > > >> > > > topic has been deleted.
> > > >> > > > Manually deleting the log files will resolve the issue.
> > > >> > > > Should
> > > >> broker
> > > >> > > > ignore log files whose topic has been deleted?
> > > >> > > >
> > > >> > > > Regards,
> > > >> > > >
> > > >> > > > Libo
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: delete a topic
Posted by Jun Rao <ju...@gmail.com>.
Search for *-ConsumerLag under kafka.server. There is a jmx bean per
partition in the follower.
Thanks,
Jun
On Fri, Aug 23, 2013 at 7:59 PM, Vadim Keylis <vk...@gmail.com> wrote:
> Hi Neha. What are jmx properties to monitor for lag and progress of
> replica, until you release the tool?
>
> Sent from my iPhone
>
> On Aug 23, 2013, at 8:32 AM, Neha Narkhede <ne...@gmail.com>
> wrote:
>
> > By that, I meant bringing down all brokers, deleting zookeeper data and
> > kafka data and restarting the brokers. I suspect attempting a delete
> topic
> > might have caused something bad to happen on the ISR side. It will be
> great
> > if you can start clean and then see if the ISR is still an issue.
> >
> > Also, I filed a bug to include a tool that makes it easy to check the
> > replica lag and progress, while troubleshooting ISR membership issues -
> > https://issues.apache.org/jira/browse/KAFKA-1021
> >
> > Thanks,
> > Neha
> >
> >
> > On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
> >
> >> Hi Neha,
> >>
> >> "Wipe out the cluster" Do you mean you uninstall the cluster and
> reinstall
> >> it?
> >> Or you just delete all kafka data and zookeeper data for the cluster?
> >> This is not a blocking issue for us. Our blocking issue is that some
> >> broker will
> >> fall out of ISR and never get back to it. We have observed that from
> more
> >> that one clusters. I am not sure if it is related to the deleting issue.
> >>
> >> Regards,
> >>
> >> Libo
> >>
> >>
> >> -----Original Message-----
> >> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> >> Sent: Thursday, August 22, 2013 5:01 PM
> >> To: users@kafka.apache.org
> >> Subject: Re: delete a topic
> >>
> >> In production, we are not deleting topics yet. In test environments, if
> we
> >> have to delete topics, we wipe out the cluster. If this is a feature
> that
> >> most users are blocked on, I think it makes sense to prioritize
> >> https://issues.apache.org/jira/browse/KAFKA-330.
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis <vk...@gmail.com>
> >> wrote:
> >>
> >>> How do you guys delete a topic if such need arise?
> >>>
> >>>
> >>> On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
> >>> <neha.narkhede@gmail.com
> >>>> wrote:
> >>>
> >>>> I mentioned this in a different thread. I think the
> >>> 0.8.0-beta1-candidate1
> >>>> includes the script, but we have removed it later. I think when we
> >>> publish
> >>>> 0.8-final, this should be resolved.
> >>>>
> >>>> Thanks,
> >>>> Neha
> >>>>
> >>>>
> >>>> On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
> >>>> <neha.narkhede@gmail.com
> >>>>> wrote:
> >>>>
> >>>>> Vadim,
> >>>>>
> >>>>> The JIRA tracking this feature is
> >>>>> https://issues.apache.org/jira/browse/KAFKA-330.
> >>>>> Until we have this feature, we should remove any scripts that give
> >>>>> the impression of deleting a topic, as Jay suggested. I tried
> >>>>> looking under bin/ but couldn't find any delete topic or partition
> >> script.
> >>>>>
> >>>>> Can you tell us which script you are using?
> >>>>>
> >>>>> Thanks,
> >>>>> Neha
> >>>>>
> >>>>>
> >>>>> On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
> >>>>> <vkeylis2009@gmail.com
> >>>>> wrote:
> >>>>>
> >>>>>> Jay or Neha. What is the process of deleting the topic if the
> >>>>>> script included in the repository creates false impression?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Vadim
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> We should really remove the delete script from 0.8 if we plan
> >>>>>>> to
> >>>> release
> >>>>>>> without delete support. Right now it appears to work but
> >>>>>>> doesn't
> >>> which
> >>>>>> is
> >>>>>>> clearly not good.
> >>>>>>>
> >>>>>>> -Jay
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> >>>>>> neha.narkhede@gmail.com
> >>>>>>>> wrote:
> >>>>>>>
> >>>>>>>> We currently don't have the delete topic feature in Kafka
> >>>>>>>> 0.8. So
> >>>> any
> >>>>>>>> manual attempts to do so might have a negative impact on
> >>>>>> functionality.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Neha
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi team,
> >>>>>>>>>
> >>>>>>>>> When I delete a topic, the topic is deleted from zookeeper
> >>>>>>>>> but
> >>> its
> >>>>>> log
> >>>>>>>>> files are not deleted from
> >>>>>>>>> Brokers.
> >>>>>>>>>
> >>>>>>>>> When I restart a broker, the broker will try to sync the
> >>>>>>>>> log
> >>>> files
> >>>>>>> whose
> >>>>>>>>> topic has been deleted.
> >>>>>>>>> Manually deleting the log files will resolve the issue.
> >>>>>>>>> Should
> >>>>>> broker
> >>>>>>>>> ignore log files whose topic has been deleted?
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>>
> >>>>>>>>> Libo
> >>
>
Re: delete a topic
Posted by Vadim Keylis <vk...@gmail.com>.
Hi Neha. What are jmx properties to monitor for lag and progress of
replica, until you release the tool?
Sent from my iPhone
On Aug 23, 2013, at 8:32 AM, Neha Narkhede <ne...@gmail.com> wrote:
> By that, I meant bringing down all brokers, deleting zookeeper data and
> kafka data and restarting the brokers. I suspect attempting a delete topic
> might have caused something bad to happen on the ISR side. It will be great
> if you can start clean and then see if the ISR is still an issue.
>
> Also, I filed a bug to include a tool that makes it easy to check the
> replica lag and progress, while troubleshooting ISR membership issues -
> https://issues.apache.org/jira/browse/KAFKA-1021
>
> Thanks,
> Neha
>
>
> On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
>
>> Hi Neha,
>>
>> "Wipe out the cluster" Do you mean you uninstall the cluster and reinstall
>> it?
>> Or you just delete all kafka data and zookeeper data for the cluster?
>> This is not a blocking issue for us. Our blocking issue is that some
>> broker will
>> fall out of ISR and never get back to it. We have observed that from more
>> that one clusters. I am not sure if it is related to the deleting issue.
>>
>> Regards,
>>
>> Libo
>>
>>
>> -----Original Message-----
>> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
>> Sent: Thursday, August 22, 2013 5:01 PM
>> To: users@kafka.apache.org
>> Subject: Re: delete a topic
>>
>> In production, we are not deleting topics yet. In test environments, if we
>> have to delete topics, we wipe out the cluster. If this is a feature that
>> most users are blocked on, I think it makes sense to prioritize
>> https://issues.apache.org/jira/browse/KAFKA-330.
>>
>> Thanks,
>> Neha
>>
>>
>> On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis <vk...@gmail.com>
>> wrote:
>>
>>> How do you guys delete a topic if such need arise?
>>>
>>>
>>> On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
>>> <neha.narkhede@gmail.com
>>>> wrote:
>>>
>>>> I mentioned this in a different thread. I think the
>>> 0.8.0-beta1-candidate1
>>>> includes the script, but we have removed it later. I think when we
>>> publish
>>>> 0.8-final, this should be resolved.
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>>
>>>> On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
>>>> <neha.narkhede@gmail.com
>>>>> wrote:
>>>>
>>>>> Vadim,
>>>>>
>>>>> The JIRA tracking this feature is
>>>>> https://issues.apache.org/jira/browse/KAFKA-330.
>>>>> Until we have this feature, we should remove any scripts that give
>>>>> the impression of deleting a topic, as Jay suggested. I tried
>>>>> looking under bin/ but couldn't find any delete topic or partition
>> script.
>>>>>
>>>>> Can you tell us which script you are using?
>>>>>
>>>>> Thanks,
>>>>> Neha
>>>>>
>>>>>
>>>>> On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
>>>>> <vkeylis2009@gmail.com
>>>>> wrote:
>>>>>
>>>>>> Jay or Neha. What is the process of deleting the topic if the
>>>>>> script included in the repository creates false impression?
>>>>>>
>>>>>> Thanks,
>>>>>> Vadim
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>>> We should really remove the delete script from 0.8 if we plan
>>>>>>> to
>>>> release
>>>>>>> without delete support. Right now it appears to work but
>>>>>>> doesn't
>>> which
>>>>>> is
>>>>>>> clearly not good.
>>>>>>>
>>>>>>> -Jay
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
>>>>>> neha.narkhede@gmail.com
>>>>>>>> wrote:
>>>>>>>
>>>>>>>> We currently don't have the delete topic feature in Kafka
>>>>>>>> 0.8. So
>>>> any
>>>>>>>> manual attempts to do so might have a negative impact on
>>>>>> functionality.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Neha
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com>
>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi team,
>>>>>>>>>
>>>>>>>>> When I delete a topic, the topic is deleted from zookeeper
>>>>>>>>> but
>>> its
>>>>>> log
>>>>>>>>> files are not deleted from
>>>>>>>>> Brokers.
>>>>>>>>>
>>>>>>>>> When I restart a broker, the broker will try to sync the
>>>>>>>>> log
>>>> files
>>>>>>> whose
>>>>>>>>> topic has been deleted.
>>>>>>>>> Manually deleting the log files will resolve the issue.
>>>>>>>>> Should
>>>>>> broker
>>>>>>>>> ignore log files whose topic has been deleted?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Libo
>>
Re: delete a topic
Posted by Neha Narkhede <ne...@gmail.com>.
By that, I meant bringing down all brokers, deleting zookeeper data and
kafka data and restarting the brokers. I suspect attempting a delete topic
might have caused something bad to happen on the ISR side. It will be great
if you can start clean and then see if the ISR is still an issue.
Also, I filed a bug to include a tool that makes it easy to check the
replica lag and progress, while troubleshooting ISR membership issues -
https://issues.apache.org/jira/browse/KAFKA-1021
Thanks,
Neha
On Fri, Aug 23, 2013 at 6:34 AM, Yu, Libo <li...@citi.com> wrote:
> Hi Neha,
>
> "Wipe out the cluster" Do you mean you uninstall the cluster and reinstall
> it?
> Or you just delete all kafka data and zookeeper data for the cluster?
> This is not a blocking issue for us. Our blocking issue is that some
> broker will
> fall out of ISR and never get back to it. We have observed that from more
> that one clusters. I am not sure if it is related to the deleting issue.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Thursday, August 22, 2013 5:01 PM
> To: users@kafka.apache.org
> Subject: Re: delete a topic
>
> In production, we are not deleting topics yet. In test environments, if we
> have to delete topics, we wipe out the cluster. If this is a feature that
> most users are blocked on, I think it makes sense to prioritize
> https://issues.apache.org/jira/browse/KAFKA-330.
>
> Thanks,
> Neha
>
>
> On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis <vk...@gmail.com>
> wrote:
>
> > How do you guys delete a topic if such need arise?
> >
> >
> > On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
> > <neha.narkhede@gmail.com
> > >wrote:
> >
> > > I mentioned this in a different thread. I think the
> > 0.8.0-beta1-candidate1
> > > includes the script, but we have removed it later. I think when we
> > publish
> > > 0.8-final, this should be resolved.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
> > > <neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Vadim,
> > > >
> > > > The JIRA tracking this feature is
> > > > https://issues.apache.org/jira/browse/KAFKA-330.
> > > > Until we have this feature, we should remove any scripts that give
> > > > the impression of deleting a topic, as Jay suggested. I tried
> > > > looking under bin/ but couldn't find any delete topic or partition
> script.
> > > >
> > > > Can you tell us which script you are using?
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
> > > ><vkeylis2009@gmail.com
> > > >wrote:
> > > >
> > > >> Jay or Neha. What is the process of deleting the topic if the
> > > >> script included in the repository creates false impression?
> > > >>
> > > >> Thanks,
> > > >> Vadim
> > > >>
> > > >>
> > > >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com>
> > > wrote:
> > > >>
> > > >> > We should really remove the delete script from 0.8 if we plan
> > > >> > to
> > > release
> > > >> > without delete support. Right now it appears to work but
> > > >> > doesn't
> > which
> > > >> is
> > > >> > clearly not good.
> > > >> >
> > > >> > -Jay
> > > >> >
> > > >> >
> > > >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> > > >> neha.narkhede@gmail.com
> > > >> > >wrote:
> > > >> >
> > > >> > > We currently don't have the delete topic feature in Kafka
> > > >> > > 0.8. So
> > > any
> > > >> > > manual attempts to do so might have a negative impact on
> > > >> functionality.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Neha
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com>
> > > wrote:
> > > >> > >
> > > >> > > > Hi team,
> > > >> > > >
> > > >> > > > When I delete a topic, the topic is deleted from zookeeper
> > > >> > > > but
> > its
> > > >> log
> > > >> > > > files are not deleted from
> > > >> > > > Brokers.
> > > >> > > >
> > > >> > > > When I restart a broker, the broker will try to sync the
> > > >> > > > log
> > > files
> > > >> > whose
> > > >> > > > topic has been deleted.
> > > >> > > > Manually deleting the log files will resolve the issue.
> > > >> > > > Should
> > > >> broker
> > > >> > > > ignore log files whose topic has been deleted?
> > > >> > > >
> > > >> > > > Regards,
> > > >> > > >
> > > >> > > > Libo
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
RE: delete a topic
Posted by "Yu, Libo " <li...@citi.com>.
Hi Neha,
"Wipe out the cluster" Do you mean you uninstall the cluster and reinstall it?
Or you just delete all kafka data and zookeeper data for the cluster?
This is not a blocking issue for us. Our blocking issue is that some broker will
fall out of ISR and never get back to it. We have observed that from more
that one clusters. I am not sure if it is related to the deleting issue.
Regards,
Libo
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Thursday, August 22, 2013 5:01 PM
To: users@kafka.apache.org
Subject: Re: delete a topic
In production, we are not deleting topics yet. In test environments, if we have to delete topics, we wipe out the cluster. If this is a feature that most users are blocked on, I think it makes sense to prioritize https://issues.apache.org/jira/browse/KAFKA-330.
Thanks,
Neha
On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis <vk...@gmail.com> wrote:
> How do you guys delete a topic if such need arise?
>
>
> On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede
> <neha.narkhede@gmail.com
> >wrote:
>
> > I mentioned this in a different thread. I think the
> 0.8.0-beta1-candidate1
> > includes the script, but we have removed it later. I think when we
> publish
> > 0.8-final, this should be resolved.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede
> > <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Vadim,
> > >
> > > The JIRA tracking this feature is
> > > https://issues.apache.org/jira/browse/KAFKA-330.
> > > Until we have this feature, we should remove any scripts that give
> > > the impression of deleting a topic, as Jay suggested. I tried
> > > looking under bin/ but couldn't find any delete topic or partition script.
> > >
> > > Can you tell us which script you are using?
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis
> > ><vkeylis2009@gmail.com
> > >wrote:
> > >
> > >> Jay or Neha. What is the process of deleting the topic if the
> > >> script included in the repository creates false impression?
> > >>
> > >> Thanks,
> > >> Vadim
> > >>
> > >>
> > >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com>
> > wrote:
> > >>
> > >> > We should really remove the delete script from 0.8 if we plan
> > >> > to
> > release
> > >> > without delete support. Right now it appears to work but
> > >> > doesn't
> which
> > >> is
> > >> > clearly not good.
> > >> >
> > >> > -Jay
> > >> >
> > >> >
> > >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> > >> neha.narkhede@gmail.com
> > >> > >wrote:
> > >> >
> > >> > > We currently don't have the delete topic feature in Kafka
> > >> > > 0.8. So
> > any
> > >> > > manual attempts to do so might have a negative impact on
> > >> functionality.
> > >> > >
> > >> > > Thanks,
> > >> > > Neha
> > >> > >
> > >> > >
> > >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com>
> > wrote:
> > >> > >
> > >> > > > Hi team,
> > >> > > >
> > >> > > > When I delete a topic, the topic is deleted from zookeeper
> > >> > > > but
> its
> > >> log
> > >> > > > files are not deleted from
> > >> > > > Brokers.
> > >> > > >
> > >> > > > When I restart a broker, the broker will try to sync the
> > >> > > > log
> > files
> > >> > whose
> > >> > > > topic has been deleted.
> > >> > > > Manually deleting the log files will resolve the issue.
> > >> > > > Should
> > >> broker
> > >> > > > ignore log files whose topic has been deleted?
> > >> > > >
> > >> > > > Regards,
> > >> > > >
> > >> > > > Libo
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: delete a topic
Posted by Neha Narkhede <ne...@gmail.com>.
In production, we are not deleting topics yet. In test environments, if we
have to delete topics, we wipe out the cluster. If this is a feature that
most users are blocked on, I think it makes sense to prioritize
https://issues.apache.org/jira/browse/KAFKA-330.
Thanks,
Neha
On Thu, Aug 22, 2013 at 1:01 PM, Vadim Keylis <vk...@gmail.com> wrote:
> How do you guys delete a topic if such need arise?
>
>
> On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > I mentioned this in a different thread. I think the
> 0.8.0-beta1-candidate1
> > includes the script, but we have removed it later. I think when we
> publish
> > 0.8-final, this should be resolved.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Vadim,
> > >
> > > The JIRA tracking this feature is
> > > https://issues.apache.org/jira/browse/KAFKA-330.
> > > Until we have this feature, we should remove any scripts that give the
> > > impression of deleting a topic, as Jay suggested. I tried looking under
> > > bin/ but couldn't find any delete topic or partition script.
> > >
> > > Can you tell us which script you are using?
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis <vkeylis2009@gmail.com
> > >wrote:
> > >
> > >> Jay or Neha. What is the process of deleting the topic if the script
> > >> included in the repository creates false impression?
> > >>
> > >> Thanks,
> > >> Vadim
> > >>
> > >>
> > >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com>
> > wrote:
> > >>
> > >> > We should really remove the delete script from 0.8 if we plan to
> > release
> > >> > without delete support. Right now it appears to work but doesn't
> which
> > >> is
> > >> > clearly not good.
> > >> >
> > >> > -Jay
> > >> >
> > >> >
> > >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> > >> neha.narkhede@gmail.com
> > >> > >wrote:
> > >> >
> > >> > > We currently don't have the delete topic feature in Kafka 0.8. So
> > any
> > >> > > manual attempts to do so might have a negative impact on
> > >> functionality.
> > >> > >
> > >> > > Thanks,
> > >> > > Neha
> > >> > >
> > >> > >
> > >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com>
> > wrote:
> > >> > >
> > >> > > > Hi team,
> > >> > > >
> > >> > > > When I delete a topic, the topic is deleted from zookeeper but
> its
> > >> log
> > >> > > > files are not deleted from
> > >> > > > Brokers.
> > >> > > >
> > >> > > > When I restart a broker, the broker will try to sync the log
> > files
> > >> > whose
> > >> > > > topic has been deleted.
> > >> > > > Manually deleting the log files will resolve the issue. Should
> > >> broker
> > >> > > > ignore log files whose topic
> > >> > > > has been deleted?
> > >> > > >
> > >> > > > Regards,
> > >> > > >
> > >> > > > Libo
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: delete a topic
Posted by Vadim Keylis <vk...@gmail.com>.
How do you guys delete a topic if such need arise?
On Thu, Aug 22, 2013 at 12:29 PM, Neha Narkhede <ne...@gmail.com>wrote:
> I mentioned this in a different thread. I think the 0.8.0-beta1-candidate1
> includes the script, but we have removed it later. I think when we publish
> 0.8-final, this should be resolved.
>
> Thanks,
> Neha
>
>
> On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Vadim,
> >
> > The JIRA tracking this feature is
> > https://issues.apache.org/jira/browse/KAFKA-330.
> > Until we have this feature, we should remove any scripts that give the
> > impression of deleting a topic, as Jay suggested. I tried looking under
> > bin/ but couldn't find any delete topic or partition script.
> >
> > Can you tell us which script you are using?
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis <vkeylis2009@gmail.com
> >wrote:
> >
> >> Jay or Neha. What is the process of deleting the topic if the script
> >> included in the repository creates false impression?
> >>
> >> Thanks,
> >> Vadim
> >>
> >>
> >> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com>
> wrote:
> >>
> >> > We should really remove the delete script from 0.8 if we plan to
> release
> >> > without delete support. Right now it appears to work but doesn't which
> >> is
> >> > clearly not good.
> >> >
> >> > -Jay
> >> >
> >> >
> >> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
> >> neha.narkhede@gmail.com
> >> > >wrote:
> >> >
> >> > > We currently don't have the delete topic feature in Kafka 0.8. So
> any
> >> > > manual attempts to do so might have a negative impact on
> >> functionality.
> >> > >
> >> > > Thanks,
> >> > > Neha
> >> > >
> >> > >
> >> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com>
> wrote:
> >> > >
> >> > > > Hi team,
> >> > > >
> >> > > > When I delete a topic, the topic is deleted from zookeeper but its
> >> log
> >> > > > files are not deleted from
> >> > > > Brokers.
> >> > > >
> >> > > > When I restart a broker, the broker will try to sync the log
> files
> >> > whose
> >> > > > topic has been deleted.
> >> > > > Manually deleting the log files will resolve the issue. Should
> >> broker
> >> > > > ignore log files whose topic
> >> > > > has been deleted?
> >> > > >
> >> > > > Regards,
> >> > > >
> >> > > > Libo
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: delete a topic
Posted by Neha Narkhede <ne...@gmail.com>.
I mentioned this in a different thread. I think the 0.8.0-beta1-candidate1
includes the script, but we have removed it later. I think when we publish
0.8-final, this should be resolved.
Thanks,
Neha
On Thu, Aug 22, 2013 at 12:24 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Vadim,
>
> The JIRA tracking this feature is
> https://issues.apache.org/jira/browse/KAFKA-330.
> Until we have this feature, we should remove any scripts that give the
> impression of deleting a topic, as Jay suggested. I tried looking under
> bin/ but couldn't find any delete topic or partition script.
>
> Can you tell us which script you are using?
>
> Thanks,
> Neha
>
>
> On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis <vk...@gmail.com>wrote:
>
>> Jay or Neha. What is the process of deleting the topic if the script
>> included in the repository creates false impression?
>>
>> Thanks,
>> Vadim
>>
>>
>> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com> wrote:
>>
>> > We should really remove the delete script from 0.8 if we plan to release
>> > without delete support. Right now it appears to work but doesn't which
>> is
>> > clearly not good.
>> >
>> > -Jay
>> >
>> >
>> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <
>> neha.narkhede@gmail.com
>> > >wrote:
>> >
>> > > We currently don't have the delete topic feature in Kafka 0.8. So any
>> > > manual attempts to do so might have a negative impact on
>> functionality.
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > >
>> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com> wrote:
>> > >
>> > > > Hi team,
>> > > >
>> > > > When I delete a topic, the topic is deleted from zookeeper but its
>> log
>> > > > files are not deleted from
>> > > > Brokers.
>> > > >
>> > > > When I restart a broker, the broker will try to sync the log files
>> > whose
>> > > > topic has been deleted.
>> > > > Manually deleting the log files will resolve the issue. Should
>> broker
>> > > > ignore log files whose topic
>> > > > has been deleted?
>> > > >
>> > > > Regards,
>> > > >
>> > > > Libo
>> > > >
>> > > >
>> > >
>> >
>>
>
>
Re: delete a topic
Posted by Neha Narkhede <ne...@gmail.com>.
Vadim,
The JIRA tracking this feature is
https://issues.apache.org/jira/browse/KAFKA-330.
Until we have this feature, we should remove any scripts that give the
impression of deleting a topic, as Jay suggested. I tried looking under
bin/ but couldn't find any delete topic or partition script.
Can you tell us which script you are using?
Thanks,
Neha
On Thu, Aug 22, 2013 at 11:40 AM, Vadim Keylis <vk...@gmail.com>wrote:
> Jay or Neha. What is the process of deleting the topic if the script
> included in the repository creates false impression?
>
> Thanks,
> Vadim
>
>
> On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com> wrote:
>
> > We should really remove the delete script from 0.8 if we plan to release
> > without delete support. Right now it appears to work but doesn't which is
> > clearly not good.
> >
> > -Jay
> >
> >
> > On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > We currently don't have the delete topic feature in Kafka 0.8. So any
> > > manual attempts to do so might have a negative impact on functionality.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com> wrote:
> > >
> > > > Hi team,
> > > >
> > > > When I delete a topic, the topic is deleted from zookeeper but its
> log
> > > > files are not deleted from
> > > > Brokers.
> > > >
> > > > When I restart a broker, the broker will try to sync the log files
> > whose
> > > > topic has been deleted.
> > > > Manually deleting the log files will resolve the issue. Should broker
> > > > ignore log files whose topic
> > > > has been deleted?
> > > >
> > > > Regards,
> > > >
> > > > Libo
> > > >
> > > >
> > >
> >
>
RE: delete a topic
Posted by "Yu, Libo " <li...@citi.com>.
After I deleted a topic using the delete script and deleted all the log files, the brokers still
tried to sync for that topic. It seems a cluster restart is required.
Regards,
Libo
-----Original Message-----
From: Vadim Keylis [mailto:vkeylis2009@gmail.com]
Sent: Thursday, August 22, 2013 2:41 PM
To: users@kafka.apache.org
Subject: Re: delete a topic
Jay or Neha. What is the process of deleting the topic if the script included in the repository creates false impression?
Thanks,
Vadim
On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com> wrote:
> We should really remove the delete script from 0.8 if we plan to
> release without delete support. Right now it appears to work but
> doesn't which is clearly not good.
>
> -Jay
>
>
> On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede
> <neha.narkhede@gmail.com
> >wrote:
>
> > We currently don't have the delete topic feature in Kafka 0.8. So
> > any manual attempts to do so might have a negative impact on functionality.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com> wrote:
> >
> > > Hi team,
> > >
> > > When I delete a topic, the topic is deleted from zookeeper but its
> > > log files are not deleted from Brokers.
> > >
> > > When I restart a broker, the broker will try to sync the log
> > > files
> whose
> > > topic has been deleted.
> > > Manually deleting the log files will resolve the issue. Should
> > > broker ignore log files whose topic has been deleted?
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> >
>
Re: delete a topic
Posted by Vadim Keylis <vk...@gmail.com>.
Jay or Neha. What is the process of deleting the topic if the script
included in the repository creates false impression?
Thanks,
Vadim
On Thu, Aug 22, 2013 at 11:14 AM, Jay Kreps <ja...@gmail.com> wrote:
> We should really remove the delete script from 0.8 if we plan to release
> without delete support. Right now it appears to work but doesn't which is
> clearly not good.
>
> -Jay
>
>
> On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > We currently don't have the delete topic feature in Kafka 0.8. So any
> > manual attempts to do so might have a negative impact on functionality.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com> wrote:
> >
> > > Hi team,
> > >
> > > When I delete a topic, the topic is deleted from zookeeper but its log
> > > files are not deleted from
> > > Brokers.
> > >
> > > When I restart a broker, the broker will try to sync the log files
> whose
> > > topic has been deleted.
> > > Manually deleting the log files will resolve the issue. Should broker
> > > ignore log files whose topic
> > > has been deleted?
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> >
>
Re: delete a topic
Posted by Jay Kreps <ja...@gmail.com>.
We should really remove the delete script from 0.8 if we plan to release
without delete support. Right now it appears to work but doesn't which is
clearly not good.
-Jay
On Thu, Aug 22, 2013 at 10:57 AM, Neha Narkhede <ne...@gmail.com>wrote:
> We currently don't have the delete topic feature in Kafka 0.8. So any
> manual attempts to do so might have a negative impact on functionality.
>
> Thanks,
> Neha
>
>
> On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com> wrote:
>
> > Hi team,
> >
> > When I delete a topic, the topic is deleted from zookeeper but its log
> > files are not deleted from
> > Brokers.
> >
> > When I restart a broker, the broker will try to sync the log files whose
> > topic has been deleted.
> > Manually deleting the log files will resolve the issue. Should broker
> > ignore log files whose topic
> > has been deleted?
> >
> > Regards,
> >
> > Libo
> >
> >
>
Re: delete a topic
Posted by Neha Narkhede <ne...@gmail.com>.
We currently don't have the delete topic feature in Kafka 0.8. So any
manual attempts to do so might have a negative impact on functionality.
Thanks,
Neha
On Thu, Aug 22, 2013 at 10:30 AM, Yu, Libo <li...@citi.com> wrote:
> Hi team,
>
> When I delete a topic, the topic is deleted from zookeeper but its log
> files are not deleted from
> Brokers.
>
> When I restart a broker, the broker will try to sync the log files whose
> topic has been deleted.
> Manually deleting the log files will resolve the issue. Should broker
> ignore log files whose topic
> has been deleted?
>
> Regards,
>
> Libo
>
>