You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2014/11/20 20:08:47 UTC
Changing it so we do NOT archive hfiles by default
I think we should swap the default that has us archive hfiles rather than
just outright delete them when we are done with them. The current
configuration works for the minority of us who are running backup tools.
For the rest of us, our clusters are doing unnecessary extra work.
Background:
Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when we are
done with an hfile, it is moved to the 'archive' (hbase/.archive)
directory. A thread in the master then removes hfiles older than some
configured time. We do this rather than just delete hfiles to facilitate
backup tools -- let backup tools have a say in when an hfile is safe to
remove.
The subject on HBASE-5547 has it that the archiving behavior only happens
when the cluster is in 'backup mode', but as it turns out, later in the
issue discussion, the implementation becomes significantly easier if we
just always archive and that is what we ended up implementing and
committing.
These last few days, a few of us have been helping a user on a large
cluster who is (temporarily) doing loads of compactions with the replaced
hfiles being moved to hbase/.archive. The cleaning thread in master is not
working fast enough deleting the hfiles so there is buildup going on -- so
much so, its slowing the whole cluster down (NN operations over tens of
millions of files).
Any problem swapping the default and having users opt-in for archiving?
(I'd leave it as is in released software). I will also take a look at
having the cleaner thread do more work per cycle.
Thanks,
St.Ack
Re: Changing it so we do NOT archive hfiles by default
Posted by Elliott Clark <ec...@apache.org>.
I can't speak for what Stack is seeing there. But I've seen this too.
Basically master delete is sequential with one thread on one machine.
Compactions are multi-threaded on multiple machines.
On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org> wrote:
> Interesting that removing the files (which is just a metadata operation in
> the NN) is slower than writing the files with all their data in the first
> place.Is it really the NN that is the gating factor or is it the algorithm
> we have in HBase? I remember we had similar issue with the HLog removal
> where we rescan the WAL directory over and over for no good reason, and the
> nice guys from Flurry did a fix.
> We have a lot of stuff relying on this now, so it should be done
> carefully. You thinking 1.0+, or even earlier releases?
>
> -- Lars
> From: Stack <st...@duboce.net>
> To: HBase Dev List <de...@hbase.apache.org>
> Sent: Thursday, November 20, 2014 11:08 AM
> Subject: Changing it so we do NOT archive hfiles by default
>
> I think we should swap the default that has us archive hfiles rather than
> just outright delete them when we are done with them. The current
> configuration works for the minority of us who are running backup tools.
> For the rest of us, our clusters are doing unnecessary extra work.
>
> Background:
>
> Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when we are
> done with an hfile, it is moved to the 'archive' (hbase/.archive)
> directory. A thread in the master then removes hfiles older than some
> configured time. We do this rather than just delete hfiles to facilitate
> backup tools -- let backup tools have a say in when an hfile is safe to
> remove.
>
> The subject on HBASE-5547 has it that the archiving behavior only happens
> when the cluster is in 'backup mode', but as it turns out, later in the
> issue discussion, the implementation becomes significantly easier if we
> just always archive and that is what we ended up implementing and
> committing.
>
> These last few days, a few of us have been helping a user on a large
> cluster who is (temporarily) doing loads of compactions with the replaced
> hfiles being moved to hbase/.archive. The cleaning thread in master is not
> working fast enough deleting the hfiles so there is buildup going on -- so
> much so, its slowing the whole cluster down (NN operations over tens of
> millions of files).
>
> Any problem swapping the default and having users opt-in for archiving?
> (I'd leave it as is in released software). I will also take a look at
> having the cleaner thread do more work per cycle.
>
> Thanks,
> St.Ack
>
>
>
>
Re: Changing it so we do NOT archive hfiles by default
Posted by lars hofhansl <la...@apache.org>.
So it is really the NameNode that is at issue here...?
Maybe we can do something like the old behavior if the archive duration is set to 0.
-- Lars
From: Esteban Gutierrez <es...@cloudera.com>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Friday, November 21, 2014 3:16 PM
Subject: Re: Changing it so we do NOT archive hfiles by default
For the specific case Stack mentioned here there are no snapshots enabled
and its an 0.94.x release so no real need for this user to have the archive
enabled. I've also seen this issue on 0.98 on with a busy NN (deletions
pile up)
I think it should be fine to fall back to the old behavior if snapshots are
not being used and delete compacted files or HFiles from a dropped table
immediately.
One problem with HBASE-11360 was to maintain a better compatibility with
snapshots in the current they work in branch-1 with the manifest file.
cheers,
esteban.
--
Cloudera, Inc.
On Fri, Nov 21, 2014 at 2:50 PM, Dave Latham <la...@davelink.net> wrote:
> Yes, there were definitely issues with the way the file cleaners worked
> where it ends up doing NameNode lookups or even scans for every single file
> before the cleaner allows it to be removed.
> What version of HBase are these folks using? Do they have snapshots
> enabled?
> Here's tickets where we fixed a couple of these performance issues where
> the master just could not keep up at large scale:
> HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of the
> SnapshotFileCache.
> The fixes were generally to check batches of files at a time instead of
> hitting the NameNode for every file.
>
> I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
> snapshots are enabled that could be the same issue.
>
> I'd be sad to see the solution be that you can't both have snapshots or
> backups and operate at large scale.
>
> Dave
>
> On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
>
> > On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org>
> wrote:
> >
> > > Interesting that removing the files (which is just a metadata operation
> > in
> > > the NN) is slower than writing the files with all their data in the
> first
> > > place.Is it really the NN that is the gating factor or is it the
> > algorithm
> > > we have in HBase? I remember we had similar issue with the HLog removal
> > > where we rescan the WAL directory over and over for no good reason, and
> > the
> > > nice guys from Flurry did a fix.
> > >
> >
> > There is yes, an open question on why the cleaner can not keep up. Am
> > looking into this too (High level, millions of files in a single dir)
> >
> >
> >
> > > We have a lot of stuff relying on this now, so it should be done
> > > carefully. You thinking 1.0+, or even earlier releases?
> > >
> > >
> > Yes. It seems a bunch of items have come to rely on this behavior since
> it
> > was introduced way back. Was thinking 1.0, yes, but after the input here
> > and offlist by Matteo, my hope of an easy fix has taken a dent.
> >
> > Thanks for the input lads,
> > St.Ack
> >
> >
> >
> >
> > > -- Lars
> > > From: Stack <st...@duboce.net>
> > > To: HBase Dev List <de...@hbase.apache.org>
> > > Sent: Thursday, November 20, 2014 11:08 AM
> > > Subject: Changing it so we do NOT archive hfiles by default
> > >
> > > I think we should swap the default that has us archive hfiles rather
> than
> > > just outright delete them when we are done with them. The current
> > > configuration works for the minority of us who are running backup
> tools.
> > > For the rest of us, our clusters are doing unnecessary extra work.
> > >
> > > Background:
> > >
> > > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when we
> > are
> > > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > > directory. A thread in the master then removes hfiles older than some
> > > configured time. We do this rather than just delete hfiles to
> facilitate
> > > backup tools -- let backup tools have a say in when an hfile is safe to
> > > remove.
> > >
> > > The subject on HBASE-5547 has it that the archiving behavior only
> happens
> > > when the cluster is in 'backup mode', but as it turns out, later in the
> > > issue discussion, the implementation becomes significantly easier if we
> > > just always archive and that is what we ended up implementing and
> > > committing.
> > >
> > > These last few days, a few of us have been helping a user on a large
> > > cluster who is (temporarily) doing loads of compactions with the
> replaced
> > > hfiles being moved to hbase/.archive. The cleaning thread in master is
> > not
> > > working fast enough deleting the hfiles so there is buildup going on --
> > so
> > > much so, its slowing the whole cluster down (NN operations over tens of
> > > millions of files).
> > >
> > > Any problem swapping the default and having users opt-in for archiving?
> > > (I'd leave it as is in released software). I will also take a look at
> > > having the cleaner thread do more work per cycle.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > >
> > >
> > >
> >
>
Re: Changing it so we do NOT archive hfiles by default
Posted by Stack <st...@duboce.net>.
Thanks for the helpful discussion. To close out this thread, in summary,
turning off 'archiving' is a non-starter; it is a fundamental at this
stage. Let me instead work on making delete do more work per cycle over in
HBASE-12626 "Archive cleaner cannot keep up; it maxes out at about 400k
deletes/hour" (multithread it or add a bulk delete to NN as Matteo
suggests). I also opened HBASE-12627 "Add back snapshot batching facility
from HBASE-11360 dropped by HBASE-11742" (Thanks for chiming in Mr. Latham).
Let me work on the above.
Thanks,
St.Ack
P.S. For those interested, the cluster was of ~500 nodes. It was RAM
constrained; other processes on box also need RAM. Over a period of days,
the loading was thrown off kilter because it started taking double writes
going from one schema to another (Cluster was running hot before the double
loading was enabled). The single-threaded master cleaner was deleting an
archived file every 9ms on average, about 400k deletes an hour. The
constrained RAM and their having 4-5 column families had them creating
files in excess of this rate so we backed up.
On Mon, Nov 24, 2014 at 5:35 PM, lars hofhansl <la...@apache.org> wrote:
> Ah. I agree. Let's get that back in. Should not be too hard.
> As for this specific issue - apparently there are no snapshots involved.
> Are we still doing the work then? Need to study the code again.
> If it is the Namenode stuck on doing deletes, deleting the files
> immediately won't help (well it would safe a rename in the NameNode).
> Chatted with Stack a bit. Seems it takes 7-8ms for the NN to delete a file
> from a single client thread.
> The NameNode definitely can do operations per second, so doing that
> locally on each region server as soon as the HFile expires might help. Or
> maybe we need multiple threads in the archiver.
> -- Lars
> From: Dave Latham <la...@davelink.net>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Monday, November 24, 2014 9:51 AM
> Subject: Re: Changing it so we do NOT archive hfiles by default
>
> Even with the manifest feature, it is still inefficient to iterate over
> every hfile in the archive and check with the namenode to see if there is a
> new snapshot manifest that may reference that single hfile rather than
> doing that check a single time for the list of all archived hfiles.
>
>
>
> On Sat, Nov 22, 2014 at 3:08 PM, lars hofhansl <la...@apache.org> wrote:
>
> Hit send to fast.
> I meant to say: "Actually is HBASE-11360 still needed when we have
> manifests?"
> From: lars hofhansl <la...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <
> larsh@apache.org>
> Sent: Saturday, November 22, 2014 2:58 PM
> Subject: Re: Changing it so we do NOT archive hfiles by default
>
> Actually in HBASE-11360 when we have manifests? The problem was scanning
> all those reference files, now those are all replaced with a manifest so
> maybe this is not a problem.-- Lars
>
> From: lars hofhansl <la...@apache.org>
>
>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Sent: Saturday, November 22, 2014 2:41 PM
> Subject: Re: Changing it so we do NOT archive hfiles by default
>
> I agree. I did not realize we undid HBASE-11360. Offhand I see no reason
> why it had to be rolled back completely, rather than being adapted.We need
> to bring that functionality back.
> -- Lars
> From: Dave Latham <la...@davelink.net>
>
>
> To: dev@hbase.apache.org
> Sent: Saturday, November 22, 2014 7:04 AM
> Subject: Re: Changing it so we do NOT archive hfiles by default
>
> If no snapshots are enabled, then I'll definitely be curious to hear more
> on the cause of not keeping up.
> I also think it's reasonable to delete files directly if there is no use
> for them in the archive.
>
> However, HBase does need to be able to handle large scale archive cleaning
> for those who are using archive based features. One important way is
> processing the checks in batches rather than one at a time. For
> HBASE-11360 as an example, even with the manifest file there's no reason we
> can't still check the archive files against the manifests in batches rather
> than reverting it to one at a time - that part of the fix is compatible and
> still important.
>
> I hope you guys are able to get past the issue for this cluster but that we
> can also address it at large.
>
>
>
> On Fri, Nov 21, 2014 at 3:16 PM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
>
> > For the specific case Stack mentioned here there are no snapshots enabled
> > and its an 0.94.x release so no real need for this user to have the
> archive
> > enabled. I've also seen this issue on 0.98 on with a busy NN (deletions
> > pile up)
> >
> > I think it should be fine to fall back to the old behavior if snapshots
> are
> > not being used and delete compacted files or HFiles from a dropped table
> > immediately.
> >
> > One problem with HBASE-11360 was to maintain a better compatibility with
> > snapshots in the current they work in branch-1 with the manifest file.
> >
> > cheers,
> > esteban.
> >
> >
> >
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Fri, Nov 21, 2014 at 2:50 PM, Dave Latham <la...@davelink.net>
> wrote:
> >
> > > Yes, there were definitely issues with the way the file cleaners worked
> > > where it ends up doing NameNode lookups or even scans for every single
> > file
> > > before the cleaner allows it to be removed.
> > > What version of HBase are these folks using? Do they have snapshots
> > > enabled?
> > > Here's tickets where we fixed a couple of these performance issues
> where
> > > the master just could not keep up at large scale:
> > > HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of
> > the
> > > SnapshotFileCache.
> > > The fixes were generally to check batches of files at a time instead of
> > > hitting the NameNode for every file.
> > >
> > > I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
> > > snapshots are enabled that could be the same issue.
> > >
> > > I'd be sad to see the solution be that you can't both have snapshots or
> > > backups and operate at large scale.
> > >
> > > Dave
> > >
> > > On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org>
> > > wrote:
> > > >
> > > > > Interesting that removing the files (which is just a metadata
> > operation
> > > > in
> > > > > the NN) is slower than writing the files with all their data in the
> > > first
> > > > > place.Is it really the NN that is the gating factor or is it the
> > > > algorithm
> > > > > we have in HBase? I remember we had similar issue with the HLog
> > removal
> > > > > where we rescan the WAL directory over and over for no good reason,
> > and
> > > > the
> > > > > nice guys from Flurry did a fix.
> > > > >
> > > >
> > > > There is yes, an open question on why the cleaner can not keep up. Am
> > > > looking into this too (High level, millions of files in a single dir)
> > > >
> > > >
> > > >
> > > > > We have a lot of stuff relying on this now, so it should be done
> > > > > carefully. You thinking 1.0+, or even earlier releases?
> > > > >
> > > > >
> > > > Yes. It seems a bunch of items have come to rely on this behavior
> since
> > > it
> > > > was introduced way back. Was thinking 1.0, yes, but after the input
> > here
> > > > and offlist by Matteo, my hope of an easy fix has taken a dent.
> > > >
> > > > Thanks for the input lads,
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > > > -- Lars
> > > > > From: Stack <st...@duboce.net>
> > > > > To: HBase Dev List <de...@hbase.apache.org>
> > > > > Sent: Thursday, November 20, 2014 11:08 AM
> > > > > Subject: Changing it so we do NOT archive hfiles by default
> > > > >
> > > > > I think we should swap the default that has us archive hfiles
> rather
> > > than
> > > > > just outright delete them when we are done with them. The current
> > > > > configuration works for the minority of us who are running backup
> > > tools.
> > > > > For the rest of us, our clusters are doing unnecessary extra work.
> > > > >
> > > > > Background:
> > > > >
> > > > > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547),
> when
> > we
> > > > are
> > > > > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > > > > directory. A thread in the master then removes hfiles older than
> some
> > > > > configured time. We do this rather than just delete hfiles to
> > > facilitate
> > > > > backup tools -- let backup tools have a say in when an hfile is
> safe
> > to
> > > > > remove.
> > > > >
> > > > > The subject on HBASE-5547 has it that the archiving behavior only
> > > happens
> > > > > when the cluster is in 'backup mode', but as it turns out, later in
> > the
> > > > > issue discussion, the implementation becomes significantly easier
> if
> > we
> > > > > just always archive and that is what we ended up implementing and
> > > > > committing.
> > > > >
> > > > > These last few days, a few of us have been helping a user on a
> large
> > > > > cluster who is (temporarily) doing loads of compactions with the
> > > replaced
> > > > > hfiles being moved to hbase/.archive. The cleaning thread in master
> > is
> > > > not
> > > > > working fast enough deleting the hfiles so there is buildup going
> on
> > --
> > > > so
> > > > > much so, its slowing the whole cluster down (NN operations over
> tens
> > of
> > > > > millions of files).
> > > > >
> > > > > Any problem swapping the default and having users opt-in for
> > archiving?
> > > > > (I'd leave it as is in released software). I will also take a look
> > at
> > > > > having the cleaner thread do more work per cycle.
> > > > >
> > > > > Thanks,
> > > > > St.Ack
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
>
>
>
>
>
>
>
>
>
Re: Changing it so we do NOT archive hfiles by default
Posted by lars hofhansl <la...@apache.org>.
Ah. I agree. Let's get that back in. Should not be too hard.
As for this specific issue - apparently there are no snapshots involved. Are we still doing the work then? Need to study the code again.
If it is the Namenode stuck on doing deletes, deleting the files immediately won't help (well it would safe a rename in the NameNode).
Chatted with Stack a bit. Seems it takes 7-8ms for the NN to delete a file from a single client thread.
The NameNode definitely can do operations per second, so doing that locally on each region server as soon as the HFile expires might help. Or maybe we need multiple threads in the archiver.
-- Lars
From: Dave Latham <la...@davelink.net>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Monday, November 24, 2014 9:51 AM
Subject: Re: Changing it so we do NOT archive hfiles by default
Even with the manifest feature, it is still inefficient to iterate over every hfile in the archive and check with the namenode to see if there is a new snapshot manifest that may reference that single hfile rather than doing that check a single time for the list of all archived hfiles.
On Sat, Nov 22, 2014 at 3:08 PM, lars hofhansl <la...@apache.org> wrote:
Hit send to fast.
I meant to say: "Actually is HBASE-11360 still needed when we have manifests?"
From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <la...@apache.org>
Sent: Saturday, November 22, 2014 2:58 PM
Subject: Re: Changing it so we do NOT archive hfiles by default
Actually in HBASE-11360 when we have manifests? The problem was scanning all those reference files, now those are all replaced with a manifest so maybe this is not a problem.-- Lars
From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Saturday, November 22, 2014 2:41 PM
Subject: Re: Changing it so we do NOT archive hfiles by default
I agree. I did not realize we undid HBASE-11360. Offhand I see no reason why it had to be rolled back completely, rather than being adapted.We need to bring that functionality back.
-- Lars
From: Dave Latham <la...@davelink.net>
To: dev@hbase.apache.org
Sent: Saturday, November 22, 2014 7:04 AM
Subject: Re: Changing it so we do NOT archive hfiles by default
If no snapshots are enabled, then I'll definitely be curious to hear more
on the cause of not keeping up.
I also think it's reasonable to delete files directly if there is no use
for them in the archive.
However, HBase does need to be able to handle large scale archive cleaning
for those who are using archive based features. One important way is
processing the checks in batches rather than one at a time. For
HBASE-11360 as an example, even with the manifest file there's no reason we
can't still check the archive files against the manifests in batches rather
than reverting it to one at a time - that part of the fix is compatible and
still important.
I hope you guys are able to get past the issue for this cluster but that we
can also address it at large.
On Fri, Nov 21, 2014 at 3:16 PM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> For the specific case Stack mentioned here there are no snapshots enabled
> and its an 0.94.x release so no real need for this user to have the archive
> enabled. I've also seen this issue on 0.98 on with a busy NN (deletions
> pile up)
>
> I think it should be fine to fall back to the old behavior if snapshots are
> not being used and delete compacted files or HFiles from a dropped table
> immediately.
>
> One problem with HBASE-11360 was to maintain a better compatibility with
> snapshots in the current they work in branch-1 with the manifest file.
>
> cheers,
> esteban.
>
>
>
>
> --
> Cloudera, Inc.
>
>
> On Fri, Nov 21, 2014 at 2:50 PM, Dave Latham <la...@davelink.net> wrote:
>
> > Yes, there were definitely issues with the way the file cleaners worked
> > where it ends up doing NameNode lookups or even scans for every single
> file
> > before the cleaner allows it to be removed.
> > What version of HBase are these folks using? Do they have snapshots
> > enabled?
> > Here's tickets where we fixed a couple of these performance issues where
> > the master just could not keep up at large scale:
> > HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of
> the
> > SnapshotFileCache.
> > The fixes were generally to check batches of files at a time instead of
> > hitting the NameNode for every file.
> >
> > I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
> > snapshots are enabled that could be the same issue.
> >
> > I'd be sad to see the solution be that you can't both have snapshots or
> > backups and operate at large scale.
> >
> > Dave
> >
> > On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > >
> > > > Interesting that removing the files (which is just a metadata
> operation
> > > in
> > > > the NN) is slower than writing the files with all their data in the
> > first
> > > > place.Is it really the NN that is the gating factor or is it the
> > > algorithm
> > > > we have in HBase? I remember we had similar issue with the HLog
> removal
> > > > where we rescan the WAL directory over and over for no good reason,
> and
> > > the
> > > > nice guys from Flurry did a fix.
> > > >
> > >
> > > There is yes, an open question on why the cleaner can not keep up. Am
> > > looking into this too (High level, millions of files in a single dir)
> > >
> > >
> > >
> > > > We have a lot of stuff relying on this now, so it should be done
> > > > carefully. You thinking 1.0+, or even earlier releases?
> > > >
> > > >
> > > Yes. It seems a bunch of items have come to rely on this behavior since
> > it
> > > was introduced way back. Was thinking 1.0, yes, but after the input
> here
> > > and offlist by Matteo, my hope of an easy fix has taken a dent.
> > >
> > > Thanks for the input lads,
> > > St.Ack
> > >
> > >
> > >
> > >
> > > > -- Lars
> > > > From: Stack <st...@duboce.net>
> > > > To: HBase Dev List <de...@hbase.apache.org>
> > > > Sent: Thursday, November 20, 2014 11:08 AM
> > > > Subject: Changing it so we do NOT archive hfiles by default
> > > >
> > > > I think we should swap the default that has us archive hfiles rather
> > than
> > > > just outright delete them when we are done with them. The current
> > > > configuration works for the minority of us who are running backup
> > tools.
> > > > For the rest of us, our clusters are doing unnecessary extra work.
> > > >
> > > > Background:
> > > >
> > > > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when
> we
> > > are
> > > > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > > > directory. A thread in the master then removes hfiles older than some
> > > > configured time. We do this rather than just delete hfiles to
> > facilitate
> > > > backup tools -- let backup tools have a say in when an hfile is safe
> to
> > > > remove.
> > > >
> > > > The subject on HBASE-5547 has it that the archiving behavior only
> > happens
> > > > when the cluster is in 'backup mode', but as it turns out, later in
> the
> > > > issue discussion, the implementation becomes significantly easier if
> we
> > > > just always archive and that is what we ended up implementing and
> > > > committing.
> > > >
> > > > These last few days, a few of us have been helping a user on a large
> > > > cluster who is (temporarily) doing loads of compactions with the
> > replaced
> > > > hfiles being moved to hbase/.archive. The cleaning thread in master
> is
> > > not
> > > > working fast enough deleting the hfiles so there is buildup going on
> --
> > > so
> > > > much so, its slowing the whole cluster down (NN operations over tens
> of
> > > > millions of files).
> > > >
> > > > Any problem swapping the default and having users opt-in for
> archiving?
> > > > (I'd leave it as is in released software). I will also take a look
> at
> > > > having the cleaner thread do more work per cycle.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: Changing it so we do NOT archive hfiles by default
Posted by Dave Latham <la...@davelink.net>.
Even with the manifest feature, it is still inefficient to iterate over
every hfile in the archive and check with the namenode to see if there is a
new snapshot manifest that may reference that single hfile rather than
doing that check a single time for the list of all archived hfiles.
On Sat, Nov 22, 2014 at 3:08 PM, lars hofhansl <la...@apache.org> wrote:
> Hit send to fast.
> I meant to say: "Actually is HBASE-11360 still needed when we have
> manifests?"
> From: lars hofhansl <la...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <
> larsh@apache.org>
> Sent: Saturday, November 22, 2014 2:58 PM
> Subject: Re: Changing it so we do NOT archive hfiles by default
>
> Actually in HBASE-11360 when we have manifests? The problem was scanning
> all those reference files, now those are all replaced with a manifest so
> maybe this is not a problem.-- Lars
>
> From: lars hofhansl <la...@apache.org>
>
>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Sent: Saturday, November 22, 2014 2:41 PM
> Subject: Re: Changing it so we do NOT archive hfiles by default
>
> I agree. I did not realize we undid HBASE-11360. Offhand I see no reason
> why it had to be rolled back completely, rather than being adapted.We need
> to bring that functionality back.
> -- Lars
> From: Dave Latham <la...@davelink.net>
>
>
> To: dev@hbase.apache.org
> Sent: Saturday, November 22, 2014 7:04 AM
> Subject: Re: Changing it so we do NOT archive hfiles by default
>
> If no snapshots are enabled, then I'll definitely be curious to hear more
> on the cause of not keeping up.
> I also think it's reasonable to delete files directly if there is no use
> for them in the archive.
>
> However, HBase does need to be able to handle large scale archive cleaning
> for those who are using archive based features. One important way is
> processing the checks in batches rather than one at a time. For
> HBASE-11360 as an example, even with the manifest file there's no reason we
> can't still check the archive files against the manifests in batches rather
> than reverting it to one at a time - that part of the fix is compatible and
> still important.
>
> I hope you guys are able to get past the issue for this cluster but that we
> can also address it at large.
>
>
>
> On Fri, Nov 21, 2014 at 3:16 PM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
>
> > For the specific case Stack mentioned here there are no snapshots enabled
> > and its an 0.94.x release so no real need for this user to have the
> archive
> > enabled. I've also seen this issue on 0.98 on with a busy NN (deletions
> > pile up)
> >
> > I think it should be fine to fall back to the old behavior if snapshots
> are
> > not being used and delete compacted files or HFiles from a dropped table
> > immediately.
> >
> > One problem with HBASE-11360 was to maintain a better compatibility with
> > snapshots in the current they work in branch-1 with the manifest file.
> >
> > cheers,
> > esteban.
> >
> >
> >
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Fri, Nov 21, 2014 at 2:50 PM, Dave Latham <la...@davelink.net>
> wrote:
> >
> > > Yes, there were definitely issues with the way the file cleaners worked
> > > where it ends up doing NameNode lookups or even scans for every single
> > file
> > > before the cleaner allows it to be removed.
> > > What version of HBase are these folks using? Do they have snapshots
> > > enabled?
> > > Here's tickets where we fixed a couple of these performance issues
> where
> > > the master just could not keep up at large scale:
> > > HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of
> > the
> > > SnapshotFileCache.
> > > The fixes were generally to check batches of files at a time instead of
> > > hitting the NameNode for every file.
> > >
> > > I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
> > > snapshots are enabled that could be the same issue.
> > >
> > > I'd be sad to see the solution be that you can't both have snapshots or
> > > backups and operate at large scale.
> > >
> > > Dave
> > >
> > > On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org>
> > > wrote:
> > > >
> > > > > Interesting that removing the files (which is just a metadata
> > operation
> > > > in
> > > > > the NN) is slower than writing the files with all their data in the
> > > first
> > > > > place.Is it really the NN that is the gating factor or is it the
> > > > algorithm
> > > > > we have in HBase? I remember we had similar issue with the HLog
> > removal
> > > > > where we rescan the WAL directory over and over for no good reason,
> > and
> > > > the
> > > > > nice guys from Flurry did a fix.
> > > > >
> > > >
> > > > There is yes, an open question on why the cleaner can not keep up. Am
> > > > looking into this too (High level, millions of files in a single dir)
> > > >
> > > >
> > > >
> > > > > We have a lot of stuff relying on this now, so it should be done
> > > > > carefully. You thinking 1.0+, or even earlier releases?
> > > > >
> > > > >
> > > > Yes. It seems a bunch of items have come to rely on this behavior
> since
> > > it
> > > > was introduced way back. Was thinking 1.0, yes, but after the input
> > here
> > > > and offlist by Matteo, my hope of an easy fix has taken a dent.
> > > >
> > > > Thanks for the input lads,
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > > > -- Lars
> > > > > From: Stack <st...@duboce.net>
> > > > > To: HBase Dev List <de...@hbase.apache.org>
> > > > > Sent: Thursday, November 20, 2014 11:08 AM
> > > > > Subject: Changing it so we do NOT archive hfiles by default
> > > > >
> > > > > I think we should swap the default that has us archive hfiles
> rather
> > > than
> > > > > just outright delete them when we are done with them. The current
> > > > > configuration works for the minority of us who are running backup
> > > tools.
> > > > > For the rest of us, our clusters are doing unnecessary extra work.
> > > > >
> > > > > Background:
> > > > >
> > > > > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547),
> when
> > we
> > > > are
> > > > > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > > > > directory. A thread in the master then removes hfiles older than
> some
> > > > > configured time. We do this rather than just delete hfiles to
> > > facilitate
> > > > > backup tools -- let backup tools have a say in when an hfile is
> safe
> > to
> > > > > remove.
> > > > >
> > > > > The subject on HBASE-5547 has it that the archiving behavior only
> > > happens
> > > > > when the cluster is in 'backup mode', but as it turns out, later in
> > the
> > > > > issue discussion, the implementation becomes significantly easier
> if
> > we
> > > > > just always archive and that is what we ended up implementing and
> > > > > committing.
> > > > >
> > > > > These last few days, a few of us have been helping a user on a
> large
> > > > > cluster who is (temporarily) doing loads of compactions with the
> > > replaced
> > > > > hfiles being moved to hbase/.archive. The cleaning thread in master
> > is
> > > > not
> > > > > working fast enough deleting the hfiles so there is buildup going
> on
> > --
> > > > so
> > > > > much so, its slowing the whole cluster down (NN operations over
> tens
> > of
> > > > > millions of files).
> > > > >
> > > > > Any problem swapping the default and having users opt-in for
> > archiving?
> > > > > (I'd leave it as is in released software). I will also take a look
> > at
> > > > > having the cleaner thread do more work per cycle.
> > > > >
> > > > > Thanks,
> > > > > St.Ack
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
>
>
>
>
>
Re: Changing it so we do NOT archive hfiles by default
Posted by lars hofhansl <la...@apache.org>.
Hit send to fast.
I meant to say: "Actually is HBASE-11360 still needed when we have manifests?"
From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <la...@apache.org>
Sent: Saturday, November 22, 2014 2:58 PM
Subject: Re: Changing it so we do NOT archive hfiles by default
Actually in HBASE-11360 when we have manifests? The problem was scanning all those reference files, now those are all replaced with a manifest so maybe this is not a problem.-- Lars
From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Saturday, November 22, 2014 2:41 PM
Subject: Re: Changing it so we do NOT archive hfiles by default
I agree. I did not realize we undid HBASE-11360. Offhand I see no reason why it had to be rolled back completely, rather than being adapted.We need to bring that functionality back.
-- Lars
From: Dave Latham <la...@davelink.net>
To: dev@hbase.apache.org
Sent: Saturday, November 22, 2014 7:04 AM
Subject: Re: Changing it so we do NOT archive hfiles by default
If no snapshots are enabled, then I'll definitely be curious to hear more
on the cause of not keeping up.
I also think it's reasonable to delete files directly if there is no use
for them in the archive.
However, HBase does need to be able to handle large scale archive cleaning
for those who are using archive based features. One important way is
processing the checks in batches rather than one at a time. For
HBASE-11360 as an example, even with the manifest file there's no reason we
can't still check the archive files against the manifests in batches rather
than reverting it to one at a time - that part of the fix is compatible and
still important.
I hope you guys are able to get past the issue for this cluster but that we
can also address it at large.
On Fri, Nov 21, 2014 at 3:16 PM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> For the specific case Stack mentioned here there are no snapshots enabled
> and its an 0.94.x release so no real need for this user to have the archive
> enabled. I've also seen this issue on 0.98 on with a busy NN (deletions
> pile up)
>
> I think it should be fine to fall back to the old behavior if snapshots are
> not being used and delete compacted files or HFiles from a dropped table
> immediately.
>
> One problem with HBASE-11360 was to maintain a better compatibility with
> snapshots in the current they work in branch-1 with the manifest file.
>
> cheers,
> esteban.
>
>
>
>
> --
> Cloudera, Inc.
>
>
> On Fri, Nov 21, 2014 at 2:50 PM, Dave Latham <la...@davelink.net> wrote:
>
> > Yes, there were definitely issues with the way the file cleaners worked
> > where it ends up doing NameNode lookups or even scans for every single
> file
> > before the cleaner allows it to be removed.
> > What version of HBase are these folks using? Do they have snapshots
> > enabled?
> > Here's tickets where we fixed a couple of these performance issues where
> > the master just could not keep up at large scale:
> > HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of
> the
> > SnapshotFileCache.
> > The fixes were generally to check batches of files at a time instead of
> > hitting the NameNode for every file.
> >
> > I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
> > snapshots are enabled that could be the same issue.
> >
> > I'd be sad to see the solution be that you can't both have snapshots or
> > backups and operate at large scale.
> >
> > Dave
> >
> > On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > >
> > > > Interesting that removing the files (which is just a metadata
> operation
> > > in
> > > > the NN) is slower than writing the files with all their data in the
> > first
> > > > place.Is it really the NN that is the gating factor or is it the
> > > algorithm
> > > > we have in HBase? I remember we had similar issue with the HLog
> removal
> > > > where we rescan the WAL directory over and over for no good reason,
> and
> > > the
> > > > nice guys from Flurry did a fix.
> > > >
> > >
> > > There is yes, an open question on why the cleaner can not keep up. Am
> > > looking into this too (High level, millions of files in a single dir)
> > >
> > >
> > >
> > > > We have a lot of stuff relying on this now, so it should be done
> > > > carefully. You thinking 1.0+, or even earlier releases?
> > > >
> > > >
> > > Yes. It seems a bunch of items have come to rely on this behavior since
> > it
> > > was introduced way back. Was thinking 1.0, yes, but after the input
> here
> > > and offlist by Matteo, my hope of an easy fix has taken a dent.
> > >
> > > Thanks for the input lads,
> > > St.Ack
> > >
> > >
> > >
> > >
> > > > -- Lars
> > > > From: Stack <st...@duboce.net>
> > > > To: HBase Dev List <de...@hbase.apache.org>
> > > > Sent: Thursday, November 20, 2014 11:08 AM
> > > > Subject: Changing it so we do NOT archive hfiles by default
> > > >
> > > > I think we should swap the default that has us archive hfiles rather
> > than
> > > > just outright delete them when we are done with them. The current
> > > > configuration works for the minority of us who are running backup
> > tools.
> > > > For the rest of us, our clusters are doing unnecessary extra work.
> > > >
> > > > Background:
> > > >
> > > > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when
> we
> > > are
> > > > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > > > directory. A thread in the master then removes hfiles older than some
> > > > configured time. We do this rather than just delete hfiles to
> > facilitate
> > > > backup tools -- let backup tools have a say in when an hfile is safe
> to
> > > > remove.
> > > >
> > > > The subject on HBASE-5547 has it that the archiving behavior only
> > happens
> > > > when the cluster is in 'backup mode', but as it turns out, later in
> the
> > > > issue discussion, the implementation becomes significantly easier if
> we
> > > > just always archive and that is what we ended up implementing and
> > > > committing.
> > > >
> > > > These last few days, a few of us have been helping a user on a large
> > > > cluster who is (temporarily) doing loads of compactions with the
> > replaced
> > > > hfiles being moved to hbase/.archive. The cleaning thread in master
> is
> > > not
> > > > working fast enough deleting the hfiles so there is buildup going on
> --
> > > so
> > > > much so, its slowing the whole cluster down (NN operations over tens
> of
> > > > millions of files).
> > > >
> > > > Any problem swapping the default and having users opt-in for
> archiving?
> > > > (I'd leave it as is in released software). I will also take a look
> at
> > > > having the cleaner thread do more work per cycle.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: Changing it so we do NOT archive hfiles by default
Posted by lars hofhansl <la...@apache.org>.
Actually in HBASE-11360 when we have manifests? The problem was scanning all those reference files, now those are all replaced with a manifest so maybe this is not a problem.-- Lars
From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Saturday, November 22, 2014 2:41 PM
Subject: Re: Changing it so we do NOT archive hfiles by default
I agree. I did not realize we undid HBASE-11360. Offhand I see no reason why it had to be rolled back completely, rather than being adapted.We need to bring that functionality back.
-- Lars
From: Dave Latham <la...@davelink.net>
To: dev@hbase.apache.org
Sent: Saturday, November 22, 2014 7:04 AM
Subject: Re: Changing it so we do NOT archive hfiles by default
If no snapshots are enabled, then I'll definitely be curious to hear more
on the cause of not keeping up.
I also think it's reasonable to delete files directly if there is no use
for them in the archive.
However, HBase does need to be able to handle large scale archive cleaning
for those who are using archive based features. One important way is
processing the checks in batches rather than one at a time. For
HBASE-11360 as an example, even with the manifest file there's no reason we
can't still check the archive files against the manifests in batches rather
than reverting it to one at a time - that part of the fix is compatible and
still important.
I hope you guys are able to get past the issue for this cluster but that we
can also address it at large.
On Fri, Nov 21, 2014 at 3:16 PM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> For the specific case Stack mentioned here there are no snapshots enabled
> and its an 0.94.x release so no real need for this user to have the archive
> enabled. I've also seen this issue on 0.98 on with a busy NN (deletions
> pile up)
>
> I think it should be fine to fall back to the old behavior if snapshots are
> not being used and delete compacted files or HFiles from a dropped table
> immediately.
>
> One problem with HBASE-11360 was to maintain a better compatibility with
> snapshots in the current they work in branch-1 with the manifest file.
>
> cheers,
> esteban.
>
>
>
>
> --
> Cloudera, Inc.
>
>
> On Fri, Nov 21, 2014 at 2:50 PM, Dave Latham <la...@davelink.net> wrote:
>
> > Yes, there were definitely issues with the way the file cleaners worked
> > where it ends up doing NameNode lookups or even scans for every single
> file
> > before the cleaner allows it to be removed.
> > What version of HBase are these folks using? Do they have snapshots
> > enabled?
> > Here's tickets where we fixed a couple of these performance issues where
> > the master just could not keep up at large scale:
> > HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of
> the
> > SnapshotFileCache.
> > The fixes were generally to check batches of files at a time instead of
> > hitting the NameNode for every file.
> >
> > I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
> > snapshots are enabled that could be the same issue.
> >
> > I'd be sad to see the solution be that you can't both have snapshots or
> > backups and operate at large scale.
> >
> > Dave
> >
> > On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > >
> > > > Interesting that removing the files (which is just a metadata
> operation
> > > in
> > > > the NN) is slower than writing the files with all their data in the
> > first
> > > > place.Is it really the NN that is the gating factor or is it the
> > > algorithm
> > > > we have in HBase? I remember we had similar issue with the HLog
> removal
> > > > where we rescan the WAL directory over and over for no good reason,
> and
> > > the
> > > > nice guys from Flurry did a fix.
> > > >
> > >
> > > There is yes, an open question on why the cleaner can not keep up. Am
> > > looking into this too (High level, millions of files in a single dir)
> > >
> > >
> > >
> > > > We have a lot of stuff relying on this now, so it should be done
> > > > carefully. You thinking 1.0+, or even earlier releases?
> > > >
> > > >
> > > Yes. It seems a bunch of items have come to rely on this behavior since
> > it
> > > was introduced way back. Was thinking 1.0, yes, but after the input
> here
> > > and offlist by Matteo, my hope of an easy fix has taken a dent.
> > >
> > > Thanks for the input lads,
> > > St.Ack
> > >
> > >
> > >
> > >
> > > > -- Lars
> > > > From: Stack <st...@duboce.net>
> > > > To: HBase Dev List <de...@hbase.apache.org>
> > > > Sent: Thursday, November 20, 2014 11:08 AM
> > > > Subject: Changing it so we do NOT archive hfiles by default
> > > >
> > > > I think we should swap the default that has us archive hfiles rather
> > than
> > > > just outright delete them when we are done with them. The current
> > > > configuration works for the minority of us who are running backup
> > tools.
> > > > For the rest of us, our clusters are doing unnecessary extra work.
> > > >
> > > > Background:
> > > >
> > > > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when
> we
> > > are
> > > > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > > > directory. A thread in the master then removes hfiles older than some
> > > > configured time. We do this rather than just delete hfiles to
> > facilitate
> > > > backup tools -- let backup tools have a say in when an hfile is safe
> to
> > > > remove.
> > > >
> > > > The subject on HBASE-5547 has it that the archiving behavior only
> > happens
> > > > when the cluster is in 'backup mode', but as it turns out, later in
> the
> > > > issue discussion, the implementation becomes significantly easier if
> we
> > > > just always archive and that is what we ended up implementing and
> > > > committing.
> > > >
> > > > These last few days, a few of us have been helping a user on a large
> > > > cluster who is (temporarily) doing loads of compactions with the
> > replaced
> > > > hfiles being moved to hbase/.archive. The cleaning thread in master
> is
> > > not
> > > > working fast enough deleting the hfiles so there is buildup going on
> --
> > > so
> > > > much so, its slowing the whole cluster down (NN operations over tens
> of
> > > > millions of files).
> > > >
> > > > Any problem swapping the default and having users opt-in for
> archiving?
> > > > (I'd leave it as is in released software). I will also take a look
> at
> > > > having the cleaner thread do more work per cycle.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: Changing it so we do NOT archive hfiles by default
Posted by lars hofhansl <la...@apache.org>.
I agree. I did not realize we undid HBASE-11360. Offhand I see no reason why it had to be rolled back completely, rather than being adapted.We need to bring that functionality back.
-- Lars
From: Dave Latham <la...@davelink.net>
To: dev@hbase.apache.org
Sent: Saturday, November 22, 2014 7:04 AM
Subject: Re: Changing it so we do NOT archive hfiles by default
If no snapshots are enabled, then I'll definitely be curious to hear more
on the cause of not keeping up.
I also think it's reasonable to delete files directly if there is no use
for them in the archive.
However, HBase does need to be able to handle large scale archive cleaning
for those who are using archive based features. One important way is
processing the checks in batches rather than one at a time. For
HBASE-11360 as an example, even with the manifest file there's no reason we
can't still check the archive files against the manifests in batches rather
than reverting it to one at a time - that part of the fix is compatible and
still important.
I hope you guys are able to get past the issue for this cluster but that we
can also address it at large.
On Fri, Nov 21, 2014 at 3:16 PM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> For the specific case Stack mentioned here there are no snapshots enabled
> and its an 0.94.x release so no real need for this user to have the archive
> enabled. I've also seen this issue on 0.98 on with a busy NN (deletions
> pile up)
>
> I think it should be fine to fall back to the old behavior if snapshots are
> not being used and delete compacted files or HFiles from a dropped table
> immediately.
>
> One problem with HBASE-11360 was to maintain a better compatibility with
> snapshots in the current they work in branch-1 with the manifest file.
>
> cheers,
> esteban.
>
>
>
>
> --
> Cloudera, Inc.
>
>
> On Fri, Nov 21, 2014 at 2:50 PM, Dave Latham <la...@davelink.net> wrote:
>
> > Yes, there were definitely issues with the way the file cleaners worked
> > where it ends up doing NameNode lookups or even scans for every single
> file
> > before the cleaner allows it to be removed.
> > What version of HBase are these folks using? Do they have snapshots
> > enabled?
> > Here's tickets where we fixed a couple of these performance issues where
> > the master just could not keep up at large scale:
> > HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of
> the
> > SnapshotFileCache.
> > The fixes were generally to check batches of files at a time instead of
> > hitting the NameNode for every file.
> >
> > I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
> > snapshots are enabled that could be the same issue.
> >
> > I'd be sad to see the solution be that you can't both have snapshots or
> > backups and operate at large scale.
> >
> > Dave
> >
> > On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > >
> > > > Interesting that removing the files (which is just a metadata
> operation
> > > in
> > > > the NN) is slower than writing the files with all their data in the
> > first
> > > > place.Is it really the NN that is the gating factor or is it the
> > > algorithm
> > > > we have in HBase? I remember we had similar issue with the HLog
> removal
> > > > where we rescan the WAL directory over and over for no good reason,
> and
> > > the
> > > > nice guys from Flurry did a fix.
> > > >
> > >
> > > There is yes, an open question on why the cleaner can not keep up. Am
> > > looking into this too (High level, millions of files in a single dir)
> > >
> > >
> > >
> > > > We have a lot of stuff relying on this now, so it should be done
> > > > carefully. You thinking 1.0+, or even earlier releases?
> > > >
> > > >
> > > Yes. It seems a bunch of items have come to rely on this behavior since
> > it
> > > was introduced way back. Was thinking 1.0, yes, but after the input
> here
> > > and offlist by Matteo, my hope of an easy fix has taken a dent.
> > >
> > > Thanks for the input lads,
> > > St.Ack
> > >
> > >
> > >
> > >
> > > > -- Lars
> > > > From: Stack <st...@duboce.net>
> > > > To: HBase Dev List <de...@hbase.apache.org>
> > > > Sent: Thursday, November 20, 2014 11:08 AM
> > > > Subject: Changing it so we do NOT archive hfiles by default
> > > >
> > > > I think we should swap the default that has us archive hfiles rather
> > than
> > > > just outright delete them when we are done with them. The current
> > > > configuration works for the minority of us who are running backup
> > tools.
> > > > For the rest of us, our clusters are doing unnecessary extra work.
> > > >
> > > > Background:
> > > >
> > > > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when
> we
> > > are
> > > > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > > > directory. A thread in the master then removes hfiles older than some
> > > > configured time. We do this rather than just delete hfiles to
> > facilitate
> > > > backup tools -- let backup tools have a say in when an hfile is safe
> to
> > > > remove.
> > > >
> > > > The subject on HBASE-5547 has it that the archiving behavior only
> > happens
> > > > when the cluster is in 'backup mode', but as it turns out, later in
> the
> > > > issue discussion, the implementation becomes significantly easier if
> we
> > > > just always archive and that is what we ended up implementing and
> > > > committing.
> > > >
> > > > These last few days, a few of us have been helping a user on a large
> > > > cluster who is (temporarily) doing loads of compactions with the
> > replaced
> > > > hfiles being moved to hbase/.archive. The cleaning thread in master
> is
> > > not
> > > > working fast enough deleting the hfiles so there is buildup going on
> --
> > > so
> > > > much so, its slowing the whole cluster down (NN operations over tens
> of
> > > > millions of files).
> > > >
> > > > Any problem swapping the default and having users opt-in for
> archiving?
> > > > (I'd leave it as is in released software). I will also take a look
> at
> > > > having the cleaner thread do more work per cycle.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: Changing it so we do NOT archive hfiles by default
Posted by Dave Latham <la...@davelink.net>.
If no snapshots are enabled, then I'll definitely be curious to hear more
on the cause of not keeping up.
I also think it's reasonable to delete files directly if there is no use
for them in the archive.
However, HBase does need to be able to handle large scale archive cleaning
for those who are using archive based features. One important way is
processing the checks in batches rather than one at a time. For
HBASE-11360 as an example, even with the manifest file there's no reason we
can't still check the archive files against the manifests in batches rather
than reverting it to one at a time - that part of the fix is compatible and
still important.
I hope you guys are able to get past the issue for this cluster but that we
can also address it at large.
On Fri, Nov 21, 2014 at 3:16 PM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> For the specific case Stack mentioned here there are no snapshots enabled
> and its an 0.94.x release so no real need for this user to have the archive
> enabled. I've also seen this issue on 0.98 on with a busy NN (deletions
> pile up)
>
> I think it should be fine to fall back to the old behavior if snapshots are
> not being used and delete compacted files or HFiles from a dropped table
> immediately.
>
> One problem with HBASE-11360 was to maintain a better compatibility with
> snapshots in the current they work in branch-1 with the manifest file.
>
> cheers,
> esteban.
>
>
>
>
> --
> Cloudera, Inc.
>
>
> On Fri, Nov 21, 2014 at 2:50 PM, Dave Latham <la...@davelink.net> wrote:
>
> > Yes, there were definitely issues with the way the file cleaners worked
> > where it ends up doing NameNode lookups or even scans for every single
> file
> > before the cleaner allows it to be removed.
> > What version of HBase are these folks using? Do they have snapshots
> > enabled?
> > Here's tickets where we fixed a couple of these performance issues where
> > the master just could not keep up at large scale:
> > HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of
> the
> > SnapshotFileCache.
> > The fixes were generally to check batches of files at a time instead of
> > hitting the NameNode for every file.
> >
> > I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
> > snapshots are enabled that could be the same issue.
> >
> > I'd be sad to see the solution be that you can't both have snapshots or
> > backups and operate at large scale.
> >
> > Dave
> >
> > On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > >
> > > > Interesting that removing the files (which is just a metadata
> operation
> > > in
> > > > the NN) is slower than writing the files with all their data in the
> > first
> > > > place.Is it really the NN that is the gating factor or is it the
> > > algorithm
> > > > we have in HBase? I remember we had similar issue with the HLog
> removal
> > > > where we rescan the WAL directory over and over for no good reason,
> and
> > > the
> > > > nice guys from Flurry did a fix.
> > > >
> > >
> > > There is yes, an open question on why the cleaner can not keep up. Am
> > > looking into this too (High level, millions of files in a single dir)
> > >
> > >
> > >
> > > > We have a lot of stuff relying on this now, so it should be done
> > > > carefully. You thinking 1.0+, or even earlier releases?
> > > >
> > > >
> > > Yes. It seems a bunch of items have come to rely on this behavior since
> > it
> > > was introduced way back. Was thinking 1.0, yes, but after the input
> here
> > > and offlist by Matteo, my hope of an easy fix has taken a dent.
> > >
> > > Thanks for the input lads,
> > > St.Ack
> > >
> > >
> > >
> > >
> > > > -- Lars
> > > > From: Stack <st...@duboce.net>
> > > > To: HBase Dev List <de...@hbase.apache.org>
> > > > Sent: Thursday, November 20, 2014 11:08 AM
> > > > Subject: Changing it so we do NOT archive hfiles by default
> > > >
> > > > I think we should swap the default that has us archive hfiles rather
> > than
> > > > just outright delete them when we are done with them. The current
> > > > configuration works for the minority of us who are running backup
> > tools.
> > > > For the rest of us, our clusters are doing unnecessary extra work.
> > > >
> > > > Background:
> > > >
> > > > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when
> we
> > > are
> > > > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > > > directory. A thread in the master then removes hfiles older than some
> > > > configured time. We do this rather than just delete hfiles to
> > facilitate
> > > > backup tools -- let backup tools have a say in when an hfile is safe
> to
> > > > remove.
> > > >
> > > > The subject on HBASE-5547 has it that the archiving behavior only
> > happens
> > > > when the cluster is in 'backup mode', but as it turns out, later in
> the
> > > > issue discussion, the implementation becomes significantly easier if
> we
> > > > just always archive and that is what we ended up implementing and
> > > > committing.
> > > >
> > > > These last few days, a few of us have been helping a user on a large
> > > > cluster who is (temporarily) doing loads of compactions with the
> > replaced
> > > > hfiles being moved to hbase/.archive. The cleaning thread in master
> is
> > > not
> > > > working fast enough deleting the hfiles so there is buildup going on
> --
> > > so
> > > > much so, its slowing the whole cluster down (NN operations over tens
> of
> > > > millions of files).
> > > >
> > > > Any problem swapping the default and having users opt-in for
> archiving?
> > > > (I'd leave it as is in released software). I will also take a look
> at
> > > > having the cleaner thread do more work per cycle.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: Changing it so we do NOT archive hfiles by default
Posted by Esteban Gutierrez <es...@cloudera.com>.
For the specific case Stack mentioned here there are no snapshots enabled
and its an 0.94.x release so no real need for this user to have the archive
enabled. I've also seen this issue on 0.98 on with a busy NN (deletions
pile up)
I think it should be fine to fall back to the old behavior if snapshots are
not being used and delete compacted files or HFiles from a dropped table
immediately.
One problem with HBASE-11360 was to maintain a better compatibility with
snapshots in the current they work in branch-1 with the manifest file.
cheers,
esteban.
--
Cloudera, Inc.
On Fri, Nov 21, 2014 at 2:50 PM, Dave Latham <la...@davelink.net> wrote:
> Yes, there were definitely issues with the way the file cleaners worked
> where it ends up doing NameNode lookups or even scans for every single file
> before the cleaner allows it to be removed.
> What version of HBase are these folks using? Do they have snapshots
> enabled?
> Here's tickets where we fixed a couple of these performance issues where
> the master just could not keep up at large scale:
> HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of the
> SnapshotFileCache.
> The fixes were generally to check batches of files at a time instead of
> hitting the NameNode for every file.
>
> I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
> snapshots are enabled that could be the same issue.
>
> I'd be sad to see the solution be that you can't both have snapshots or
> backups and operate at large scale.
>
> Dave
>
> On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
>
> > On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org>
> wrote:
> >
> > > Interesting that removing the files (which is just a metadata operation
> > in
> > > the NN) is slower than writing the files with all their data in the
> first
> > > place.Is it really the NN that is the gating factor or is it the
> > algorithm
> > > we have in HBase? I remember we had similar issue with the HLog removal
> > > where we rescan the WAL directory over and over for no good reason, and
> > the
> > > nice guys from Flurry did a fix.
> > >
> >
> > There is yes, an open question on why the cleaner can not keep up. Am
> > looking into this too (High level, millions of files in a single dir)
> >
> >
> >
> > > We have a lot of stuff relying on this now, so it should be done
> > > carefully. You thinking 1.0+, or even earlier releases?
> > >
> > >
> > Yes. It seems a bunch of items have come to rely on this behavior since
> it
> > was introduced way back. Was thinking 1.0, yes, but after the input here
> > and offlist by Matteo, my hope of an easy fix has taken a dent.
> >
> > Thanks for the input lads,
> > St.Ack
> >
> >
> >
> >
> > > -- Lars
> > > From: Stack <st...@duboce.net>
> > > To: HBase Dev List <de...@hbase.apache.org>
> > > Sent: Thursday, November 20, 2014 11:08 AM
> > > Subject: Changing it so we do NOT archive hfiles by default
> > >
> > > I think we should swap the default that has us archive hfiles rather
> than
> > > just outright delete them when we are done with them. The current
> > > configuration works for the minority of us who are running backup
> tools.
> > > For the rest of us, our clusters are doing unnecessary extra work.
> > >
> > > Background:
> > >
> > > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when we
> > are
> > > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > > directory. A thread in the master then removes hfiles older than some
> > > configured time. We do this rather than just delete hfiles to
> facilitate
> > > backup tools -- let backup tools have a say in when an hfile is safe to
> > > remove.
> > >
> > > The subject on HBASE-5547 has it that the archiving behavior only
> happens
> > > when the cluster is in 'backup mode', but as it turns out, later in the
> > > issue discussion, the implementation becomes significantly easier if we
> > > just always archive and that is what we ended up implementing and
> > > committing.
> > >
> > > These last few days, a few of us have been helping a user on a large
> > > cluster who is (temporarily) doing loads of compactions with the
> replaced
> > > hfiles being moved to hbase/.archive. The cleaning thread in master is
> > not
> > > working fast enough deleting the hfiles so there is buildup going on --
> > so
> > > much so, its slowing the whole cluster down (NN operations over tens of
> > > millions of files).
> > >
> > > Any problem swapping the default and having users opt-in for archiving?
> > > (I'd leave it as is in released software). I will also take a look at
> > > having the cleaner thread do more work per cycle.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > >
> > >
> > >
> >
>
Re: Changing it so we do NOT archive hfiles by default
Posted by Dave Latham <la...@davelink.net>.
Yes, there were definitely issues with the way the file cleaners worked
where it ends up doing NameNode lookups or even scans for every single file
before the cleaner allows it to be removed.
What version of HBase are these folks using? Do they have snapshots
enabled?
Here's tickets where we fixed a couple of these performance issues where
the master just could not keep up at large scale:
HBASE-9208 for slow ReplicationLogCleaner and HBASE-11360 for usage of the
SnapshotFileCache.
The fixes were generally to check batches of files at a time instead of
hitting the NameNode for every file.
I'm sorry to see that HBASE-11360 was reverted with HBASE-11742 so if
snapshots are enabled that could be the same issue.
I'd be sad to see the solution be that you can't both have snapshots or
backups and operate at large scale.
Dave
On Thu, Nov 20, 2014 at 12:42 PM, Stack <st...@duboce.net> wrote:
> On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org> wrote:
>
> > Interesting that removing the files (which is just a metadata operation
> in
> > the NN) is slower than writing the files with all their data in the first
> > place.Is it really the NN that is the gating factor or is it the
> algorithm
> > we have in HBase? I remember we had similar issue with the HLog removal
> > where we rescan the WAL directory over and over for no good reason, and
> the
> > nice guys from Flurry did a fix.
> >
>
> There is yes, an open question on why the cleaner can not keep up. Am
> looking into this too (High level, millions of files in a single dir)
>
>
>
> > We have a lot of stuff relying on this now, so it should be done
> > carefully. You thinking 1.0+, or even earlier releases?
> >
> >
> Yes. It seems a bunch of items have come to rely on this behavior since it
> was introduced way back. Was thinking 1.0, yes, but after the input here
> and offlist by Matteo, my hope of an easy fix has taken a dent.
>
> Thanks for the input lads,
> St.Ack
>
>
>
>
> > -- Lars
> > From: Stack <st...@duboce.net>
> > To: HBase Dev List <de...@hbase.apache.org>
> > Sent: Thursday, November 20, 2014 11:08 AM
> > Subject: Changing it so we do NOT archive hfiles by default
> >
> > I think we should swap the default that has us archive hfiles rather than
> > just outright delete them when we are done with them. The current
> > configuration works for the minority of us who are running backup tools.
> > For the rest of us, our clusters are doing unnecessary extra work.
> >
> > Background:
> >
> > Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when we
> are
> > done with an hfile, it is moved to the 'archive' (hbase/.archive)
> > directory. A thread in the master then removes hfiles older than some
> > configured time. We do this rather than just delete hfiles to facilitate
> > backup tools -- let backup tools have a say in when an hfile is safe to
> > remove.
> >
> > The subject on HBASE-5547 has it that the archiving behavior only happens
> > when the cluster is in 'backup mode', but as it turns out, later in the
> > issue discussion, the implementation becomes significantly easier if we
> > just always archive and that is what we ended up implementing and
> > committing.
> >
> > These last few days, a few of us have been helping a user on a large
> > cluster who is (temporarily) doing loads of compactions with the replaced
> > hfiles being moved to hbase/.archive. The cleaning thread in master is
> not
> > working fast enough deleting the hfiles so there is buildup going on --
> so
> > much so, its slowing the whole cluster down (NN operations over tens of
> > millions of files).
> >
> > Any problem swapping the default and having users opt-in for archiving?
> > (I'd leave it as is in released software). I will also take a look at
> > having the cleaner thread do more work per cycle.
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> >
>
Re: Changing it so we do NOT archive hfiles by default
Posted by Stack <st...@duboce.net>.
On Thu, Nov 20, 2014 at 12:21 PM, lars hofhansl <la...@apache.org> wrote:
> Interesting that removing the files (which is just a metadata operation in
> the NN) is slower than writing the files with all their data in the first
> place.Is it really the NN that is the gating factor or is it the algorithm
> we have in HBase? I remember we had similar issue with the HLog removal
> where we rescan the WAL directory over and over for no good reason, and the
> nice guys from Flurry did a fix.
>
There is yes, an open question on why the cleaner can not keep up. Am
looking into this too (High level, millions of files in a single dir)
> We have a lot of stuff relying on this now, so it should be done
> carefully. You thinking 1.0+, or even earlier releases?
>
>
Yes. It seems a bunch of items have come to rely on this behavior since it
was introduced way back. Was thinking 1.0, yes, but after the input here
and offlist by Matteo, my hope of an easy fix has taken a dent.
Thanks for the input lads,
St.Ack
> -- Lars
> From: Stack <st...@duboce.net>
> To: HBase Dev List <de...@hbase.apache.org>
> Sent: Thursday, November 20, 2014 11:08 AM
> Subject: Changing it so we do NOT archive hfiles by default
>
> I think we should swap the default that has us archive hfiles rather than
> just outright delete them when we are done with them. The current
> configuration works for the minority of us who are running backup tools.
> For the rest of us, our clusters are doing unnecessary extra work.
>
> Background:
>
> Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when we are
> done with an hfile, it is moved to the 'archive' (hbase/.archive)
> directory. A thread in the master then removes hfiles older than some
> configured time. We do this rather than just delete hfiles to facilitate
> backup tools -- let backup tools have a say in when an hfile is safe to
> remove.
>
> The subject on HBASE-5547 has it that the archiving behavior only happens
> when the cluster is in 'backup mode', but as it turns out, later in the
> issue discussion, the implementation becomes significantly easier if we
> just always archive and that is what we ended up implementing and
> committing.
>
> These last few days, a few of us have been helping a user on a large
> cluster who is (temporarily) doing loads of compactions with the replaced
> hfiles being moved to hbase/.archive. The cleaning thread in master is not
> working fast enough deleting the hfiles so there is buildup going on -- so
> much so, its slowing the whole cluster down (NN operations over tens of
> millions of files).
>
> Any problem swapping the default and having users opt-in for archiving?
> (I'd leave it as is in released software). I will also take a look at
> having the cleaner thread do more work per cycle.
>
> Thanks,
> St.Ack
>
>
>
>
Re: Changing it so we do NOT archive hfiles by default
Posted by lars hofhansl <la...@apache.org>.
Interesting that removing the files (which is just a metadata operation in the NN) is slower than writing the files with all their data in the first place.Is it really the NN that is the gating factor or is it the algorithm we have in HBase? I remember we had similar issue with the HLog removal where we rescan the WAL directory over and over for no good reason, and the nice guys from Flurry did a fix.
We have a lot of stuff relying on this now, so it should be done carefully. You thinking 1.0+, or even earlier releases?
-- Lars
From: Stack <st...@duboce.net>
To: HBase Dev List <de...@hbase.apache.org>
Sent: Thursday, November 20, 2014 11:08 AM
Subject: Changing it so we do NOT archive hfiles by default
I think we should swap the default that has us archive hfiles rather than
just outright delete them when we are done with them. The current
configuration works for the minority of us who are running backup tools.
For the rest of us, our clusters are doing unnecessary extra work.
Background:
Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when we are
done with an hfile, it is moved to the 'archive' (hbase/.archive)
directory. A thread in the master then removes hfiles older than some
configured time. We do this rather than just delete hfiles to facilitate
backup tools -- let backup tools have a say in when an hfile is safe to
remove.
The subject on HBASE-5547 has it that the archiving behavior only happens
when the cluster is in 'backup mode', but as it turns out, later in the
issue discussion, the implementation becomes significantly easier if we
just always archive and that is what we ended up implementing and
committing.
These last few days, a few of us have been helping a user on a large
cluster who is (temporarily) doing loads of compactions with the replaced
hfiles being moved to hbase/.archive. The cleaning thread in master is not
working fast enough deleting the hfiles so there is buildup going on -- so
much so, its slowing the whole cluster down (NN operations over tens of
millions of files).
Any problem swapping the default and having users opt-in for archiving?
(I'd leave it as is in released software). I will also take a look at
having the cleaner thread do more work per cycle.
Thanks,
St.Ack
Re: Changing it so we do NOT archive hfiles by default
Posted by Stack <st...@duboce.net>.
On Thu, Nov 20, 2014 at 12:05 PM, Enis Söztutar <en...@gmail.com> wrote:
> The snapshots are relying on this feature right?
Matteo just noted this off-list.
> Will we check whether any
> links back to the hfiles ?
>
>
Sorry, come again Enis... I don't follow.
> The region replicas also depend on the TTL'ed deletion of the hfiles. We do
> not explicitly create hfilelinks (on the file system), but we use hfile
> links to refer to primary region's files (either in data dir, or in
> archive).
>
Even though the TTL is default 5 minutes? You up it when region replicas?
Thanks,
St.Ack
Re: Changing it so we do NOT archive hfiles by default
Posted by Enis Söztutar <en...@gmail.com>.
The snapshots are relying on this feature right? Will we check whether any
links back to the hfiles ?
The region replicas also depend on the TTL'ed deletion of the hfiles. We do
not explicitly create hfilelinks (on the file system), but we use hfile
links to refer to primary region's files (either in data dir, or in
archive).
Enis
On Thu, Nov 20, 2014 at 11:08 AM, Stack <st...@duboce.net> wrote:
> I think we should swap the default that has us archive hfiles rather than
> just outright delete them when we are done with them. The current
> configuration works for the minority of us who are running backup tools.
> For the rest of us, our clusters are doing unnecessary extra work.
>
> Background:
>
> Since 0.94 (https://issues.apache.org/jira/browse/HBASE-5547), when we are
> done with an hfile, it is moved to the 'archive' (hbase/.archive)
> directory. A thread in the master then removes hfiles older than some
> configured time. We do this rather than just delete hfiles to facilitate
> backup tools -- let backup tools have a say in when an hfile is safe to
> remove.
>
> The subject on HBASE-5547 has it that the archiving behavior only happens
> when the cluster is in 'backup mode', but as it turns out, later in the
> issue discussion, the implementation becomes significantly easier if we
> just always archive and that is what we ended up implementing and
> committing.
>
> These last few days, a few of us have been helping a user on a large
> cluster who is (temporarily) doing loads of compactions with the replaced
> hfiles being moved to hbase/.archive. The cleaning thread in master is not
> working fast enough deleting the hfiles so there is buildup going on -- so
> much so, its slowing the whole cluster down (NN operations over tens of
> millions of files).
>
> Any problem swapping the default and having users opt-in for archiving?
> (I'd leave it as is in released software). I will also take a look at
> having the cleaner thread do more work per cycle.
>
> Thanks,
> St.Ack
>