You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Giuseppe Reina <g....@gmail.com> on 2014/07/14 18:43:17 UTC
Is it safe to delete oldWAL files?
Hi all,
after running a write intensive workload on HBase (0.98) I exhausted the
HDFS space. I realized that most of the HDFS space was occupied by files in
the path /apps/hbase/data/oldWALs/
Apparently either the old log files were not cleaned quickly enough or they
were not cleaned at all. I'm wondering if I can safely delete those files
and if I can how can I do it? Manually or is there an "hbase" command?
Also as an unrelated question, where are stored the -ROOT- and -META-
information in zookeeper?
Kind Regards
Re: Is it safe to delete oldWAL files?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
No is still there ;)
2014-07-14 14:36 GMT-04:00 Giuseppe Reina <g....@gmail.com>:
> On Mon, Jul 14, 2014 at 6:03 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > What are the values for the following config in your cluster ?
> > hbase.master.logcleaner.plugins
> > hbase.master.logcleaner.ttl
> >
> >
> They are not set, so I guess they have the default value.
>
>
>
> > BTW in 0.98, -ROOT- is gone.
> >
>
> And META?
>
>
> Thank you
>
>
>
>
> >
> > Cheers
> >
> >
> > On Mon, Jul 14, 2014 at 9:43 AM, Giuseppe Reina <g....@gmail.com>
> wrote:
> >
> > > Hi all,
> > > after running a write intensive workload on HBase (0.98) I exhausted
> > the
> > > HDFS space. I realized that most of the HDFS space was occupied by
> files
> > in
> > > the path /apps/hbase/data/oldWALs/
> > > Apparently either the old log files were not cleaned quickly enough or
> > they
> > > were not cleaned at all. I'm wondering if I can safely delete those
> > files
> > > and if I can how can I do it? Manually or is there an "hbase" command?
> > >
> > >
> > >
> > > Also as an unrelated question, where are stored the -ROOT- and -META-
> > > information in zookeeper?
> > >
> > >
> > > Kind Regards
> > >
> >
>
Re: Is it safe to delete oldWAL files?
Posted by Giuseppe Reina <g....@gmail.com>.
On Mon, Jul 14, 2014 at 6:03 PM, Ted Yu <yu...@gmail.com> wrote:
> What are the values for the following config in your cluster ?
> hbase.master.logcleaner.plugins
> hbase.master.logcleaner.ttl
>
>
They are not set, so I guess they have the default value.
> BTW in 0.98, -ROOT- is gone.
>
And META?
Thank you
>
> Cheers
>
>
> On Mon, Jul 14, 2014 at 9:43 AM, Giuseppe Reina <g....@gmail.com> wrote:
>
> > Hi all,
> > after running a write intensive workload on HBase (0.98) I exhausted
> the
> > HDFS space. I realized that most of the HDFS space was occupied by files
> in
> > the path /apps/hbase/data/oldWALs/
> > Apparently either the old log files were not cleaned quickly enough or
> they
> > were not cleaned at all. I'm wondering if I can safely delete those
> files
> > and if I can how can I do it? Manually or is there an "hbase" command?
> >
> >
> >
> > Also as an unrelated question, where are stored the -ROOT- and -META-
> > information in zookeeper?
> >
> >
> > Kind Regards
> >
>
Re: Is it safe to delete oldWAL files?
Posted by Giuseppe Reina <g....@gmail.com>.
Yes that's why I did not do it manually and I was looking for a command
On Mon, Jul 14, 2014 at 6:10 PM, Jeremy Carroll <ph...@gmail.com> wrote:
> FWIW: I would never directly delete files from the hbase.rootdir. Past
> experience has told me that if files are not being cleaned up, then there
> is either a bug, or a valid reason for keeping the files. I would dig into
> why the logcleaner is not removing the files. Could be things like
> replication between two clusters not working. I believe the default
> (depending on version) is to have the master logcleaner wake up every 10
> minutes. You should be able to set debug logging, and figure out why the
> logs are not being expunged.
>
>
> On Mon, Jul 14, 2014 at 10:03 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > What are the values for the following config in your cluster ?
> > hbase.master.logcleaner.plugins
> > hbase.master.logcleaner.ttl
> >
> > BTW in 0.98, -ROOT- is gone.
> >
> > Cheers
> >
> >
> > On Mon, Jul 14, 2014 at 9:43 AM, Giuseppe Reina <g....@gmail.com>
> wrote:
> >
> > > Hi all,
> > > after running a write intensive workload on HBase (0.98) I exhausted
> > the
> > > HDFS space. I realized that most of the HDFS space was occupied by
> files
> > in
> > > the path /apps/hbase/data/oldWALs/
> > > Apparently either the old log files were not cleaned quickly enough or
> > they
> > > were not cleaned at all. I'm wondering if I can safely delete those
> > files
> > > and if I can how can I do it? Manually or is there an "hbase" command?
> > >
> > >
> > >
> > > Also as an unrelated question, where are stored the -ROOT- and -META-
> > > information in zookeeper?
> > >
> > >
> > > Kind Regards
> > >
> >
>
Re: Is it safe to delete oldWAL files?
Posted by Jeremy Carroll <ph...@gmail.com>.
FWIW: I would never directly delete files from the hbase.rootdir. Past
experience has told me that if files are not being cleaned up, then there
is either a bug, or a valid reason for keeping the files. I would dig into
why the logcleaner is not removing the files. Could be things like
replication between two clusters not working. I believe the default
(depending on version) is to have the master logcleaner wake up every 10
minutes. You should be able to set debug logging, and figure out why the
logs are not being expunged.
On Mon, Jul 14, 2014 at 10:03 AM, Ted Yu <yu...@gmail.com> wrote:
> What are the values for the following config in your cluster ?
> hbase.master.logcleaner.plugins
> hbase.master.logcleaner.ttl
>
> BTW in 0.98, -ROOT- is gone.
>
> Cheers
>
>
> On Mon, Jul 14, 2014 at 9:43 AM, Giuseppe Reina <g....@gmail.com> wrote:
>
> > Hi all,
> > after running a write intensive workload on HBase (0.98) I exhausted
> the
> > HDFS space. I realized that most of the HDFS space was occupied by files
> in
> > the path /apps/hbase/data/oldWALs/
> > Apparently either the old log files were not cleaned quickly enough or
> they
> > were not cleaned at all. I'm wondering if I can safely delete those
> files
> > and if I can how can I do it? Manually or is there an "hbase" command?
> >
> >
> >
> > Also as an unrelated question, where are stored the -ROOT- and -META-
> > information in zookeeper?
> >
> >
> > Kind Regards
> >
>
Re: Is it safe to delete oldWAL files?
Posted by Ted Yu <yu...@gmail.com>.
What are the values for the following config in your cluster ?
hbase.master.logcleaner.plugins
hbase.master.logcleaner.ttl
BTW in 0.98, -ROOT- is gone.
Cheers
On Mon, Jul 14, 2014 at 9:43 AM, Giuseppe Reina <g....@gmail.com> wrote:
> Hi all,
> after running a write intensive workload on HBase (0.98) I exhausted the
> HDFS space. I realized that most of the HDFS space was occupied by files in
> the path /apps/hbase/data/oldWALs/
> Apparently either the old log files were not cleaned quickly enough or they
> were not cleaned at all. I'm wondering if I can safely delete those files
> and if I can how can I do it? Manually or is there an "hbase" command?
>
>
>
> Also as an unrelated question, where are stored the -ROOT- and -META-
> information in zookeeper?
>
>
> Kind Regards
>