You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shlomi Hazan <sh...@viber.com> on 2014/11/11 15:24:55 UTC
zookeeper snapshot files eat up disk space
Hi,
My zookeeper 'dataLogDir' is eating up my disk with tons of snapshot files.
what are these files? what files can I delete? are week old files
disposable?
This folder only gets bigger...
How can I avoid blowing my disk?
Thanks,
Shlomi
Re: zookeeper snapshot files eat up disk space
Posted by Ray Rodriguez <ra...@gmail.com>.
There is a sidecar jvm project called exhibitor that manages this for you. It's from Netflix so it's a bit aws-centric but still a good source for how to manage those log files. You may also want to look into some zk config settings to make sure your logs are not growing too large by truncating based on space or time.
On Tue, Nov 11, 2014 at 9:40 AM, Joe Stein <jo...@stealth.ly> wrote:
> http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Ongoing+Data+Directory+Cleanup
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
> On Tue, Nov 11, 2014 at 9:24 AM, Shlomi Hazan <sh...@viber.com> wrote:
>> Hi,
>> My zookeeper 'dataLogDir' is eating up my disk with tons of snapshot files.
>> what are these files? what files can I delete? are week old files
>> disposable?
>> This folder only gets bigger...
>> How can I avoid blowing my disk?
>> Thanks,
>> Shlomi
>>
Re: zookeeper snapshot files eat up disk space
Posted by Shlomi Hazan <sh...@viber.com>.
That looks like a complete answer.
BUT
just to be sure: it says "Automatic purging of the snapshots and
corresponding transaction logs was introduced in version 3.4.0".
using 0.8.1.1 means that I will have to purge manually, right?
Is there some convention for kafka users? e.g.: delete all but last X=3
maybe?
Shlomi
On Tue, Nov 11, 2014 at 4:27 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Ongoing+Data+Directory+Cleanup
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
> On Tue, Nov 11, 2014 at 9:24 AM, Shlomi Hazan <sh...@viber.com> wrote:
>
> > Hi,
> > My zookeeper 'dataLogDir' is eating up my disk with tons of snapshot
> files.
> > what are these files? what files can I delete? are week old files
> > disposable?
> > This folder only gets bigger...
> > How can I avoid blowing my disk?
> > Thanks,
> > Shlomi
> >
>
Re: zookeeper snapshot files eat up disk space
Posted by Joe Stein <jo...@stealth.ly>.
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#Ongoing+Data+Directory+Cleanup
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Tue, Nov 11, 2014 at 9:24 AM, Shlomi Hazan <sh...@viber.com> wrote:
> Hi,
> My zookeeper 'dataLogDir' is eating up my disk with tons of snapshot files.
> what are these files? what files can I delete? are week old files
> disposable?
> This folder only gets bigger...
> How can I avoid blowing my disk?
> Thanks,
> Shlomi
>