You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by JIEFU GONG <jg...@berkeley.edu> on 2015/07/17 01:30:26 UTC
Deleting persisted Kafka messages?
I'm running into an issue right now where after performing some tests I've
actually exhausted disk storage and am trying to clear it. This is a rather
naive question, but how would i go about manually purging my memory of all
the accumulated messages? I can't find a specific location as it seems like
I only configure the directory where logs are written, can anyone point me
in the right direction? Much appreciated!
--
Jiefu Gong
University of California, Berkeley | Class of 2017
B.A Computer Science | College of Letters and Sciences
jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
Re: Deleting persisted Kafka messages?
Posted by Foo Lim <fo...@vungle.com>.
If those messages are part of the topic that is specified in the command
line, then yes, they'll get deleted along with all the other messages in
that topic.
On Thu, Jul 16, 2015 at 4:38 PM, JIEFU GONG <jg...@berkeley.edu> wrote:
> Thanks Foo,
>
> As a follow up, since I was considering that, if I originally published
> messages under the topic 'test' with retention set to something like 168
> hours, successfully published some amount of messages to that topic (which
> are then persisted on disk), and then changed the retention.ms as you
> suggested, would that also affect the files that had been previously
> configured to be retained for 168 hours? Thanks again!
>
> On Thu, Jul 16, 2015 at 4:35 PM, Foo Lim <fo...@vungle.com> wrote:
>
> > You can set the retention.ms to something really small, i.e. 60
> seconds, &
> > kafka will delete everything older than 60 seconds (eventually).
> >
> > bin/kafka-topics.sh --zookeeper zookeeper.yoursite.com --alter --topic
> > your-topic --config retention.ms=60000
> >
> > Remember to set it back to something sane after cleanup.
> >
> > Foo
> >
> > On Thu, Jul 16, 2015 at 4:30 PM, JIEFU GONG <jg...@berkeley.edu> wrote:
> >
> > > I'm running into an issue right now where after performing some tests
> > I've
> > > actually exhausted disk storage and am trying to clear it. This is a
> > rather
> > > naive question, but how would i go about manually purging my memory of
> > all
> > > the accumulated messages? I can't find a specific location as it seems
> > like
> > > I only configure the directory where logs are written, can anyone point
> > me
> > > in the right direction? Much appreciated!
>
Re: Deleting persisted Kafka messages?
Posted by JIEFU GONG <jg...@berkeley.edu>.
Thanks Foo,
As a follow up, since I was considering that, if I originally published
messages under the topic 'test' with retention set to something like 168
hours, successfully published some amount of messages to that topic (which
are then persisted on disk), and then changed the retention.ms as you
suggested, would that also affect the files that had been previously
configured to be retained for 168 hours? Thanks again!
On Thu, Jul 16, 2015 at 4:35 PM, Foo Lim <fo...@vungle.com> wrote:
> You can set the retention.ms to something really small, i.e. 60 seconds, &
> kafka will delete everything older than 60 seconds (eventually).
>
> bin/kafka-topics.sh --zookeeper zookeeper.yoursite.com --alter --topic
> your-topic --config retention.ms=60000
>
> Remember to set it back to something sane after cleanup.
>
> Foo
>
> On Thu, Jul 16, 2015 at 4:30 PM, JIEFU GONG <jg...@berkeley.edu> wrote:
>
> > I'm running into an issue right now where after performing some tests
> I've
> > actually exhausted disk storage and am trying to clear it. This is a
> rather
> > naive question, but how would i go about manually purging my memory of
> all
> > the accumulated messages? I can't find a specific location as it seems
> like
> > I only configure the directory where logs are written, can anyone point
> me
> > in the right direction? Much appreciated!
> >
> > --
> >
> > Jiefu Gong
> > University of California, Berkeley | Class of 2017
> > B.A Computer Science | College of Letters and Sciences
> >
> > jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
> >
>
--
Jiefu Gong
University of California, Berkeley | Class of 2017
B.A Computer Science | College of Letters and Sciences
jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
Re: Deleting persisted Kafka messages?
Posted by Foo Lim <fo...@vungle.com>.
You can set the retention.ms to something really small, i.e. 60 seconds, &
kafka will delete everything older than 60 seconds (eventually).
bin/kafka-topics.sh --zookeeper zookeeper.yoursite.com --alter --topic
your-topic --config retention.ms=60000
Remember to set it back to something sane after cleanup.
Foo
On Thu, Jul 16, 2015 at 4:30 PM, JIEFU GONG <jg...@berkeley.edu> wrote:
> I'm running into an issue right now where after performing some tests I've
> actually exhausted disk storage and am trying to clear it. This is a rather
> naive question, but how would i go about manually purging my memory of all
> the accumulated messages? I can't find a specific location as it seems like
> I only configure the directory where logs are written, can anyone point me
> in the right direction? Much appreciated!
>
> --
>
> Jiefu Gong
> University of California, Berkeley | Class of 2017
> B.A Computer Science | College of Letters and Sciences
>
> jgong@berkeley.edu <el...@berkeley.edu> | (925) 400-3427
>