You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dmitriy Lyubimov <dl...@gmail.com> on 2010/09/28 04:52:23 UTC
WAL, failover questions
Hi,
i would be very grateful if somebody could clarify the following for me
please. (0.20.5)
yesterday we lost a short table (~100 rows) in production without a trace.
no matter how deep i looked in the logs of regionservers and the master, i
haven't got a clue how it might have happened.
When i looked at the table though, i did not find any files, which may mean
that it never got flushed from WAL and got compacted. When i reconstituted
it and ran compaction, the file did finally appear .
also, in the .log (hlog) directory, i noticed some duplicate entries with
both long and short host names (i.e. something like 'data4,...' and
'data4.foo.bar,...' ) which may result from the moment we decided to switch
to /etc/hosts name resolution instead of dns (just to see if that'll improve
our networking issues).
So i have the following questions :
1 -- when is WAL (or hlog, if it's the same?) is triggered and the actual
tablet file is built? is it a size-based threshold? is there an max age
threshold?
2 -- if the region server crashes, its hlog is supposed to be split and
recovered, right? is there a situation when hlog can be lost? I suppose it
doesn't matter that the region server with the same name never goes online
again (e.g. if it starts using short name instead of FQDN)?
Thanks.
-Dmitriy
Re: WAL, failover questions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Thank you, St. Ack.
by 'reconstituting' i meant table had no records, so we had to re-fill it
with the data we kept elsewhere. I couldn't find or figure any technique
that might help me to scavenge it off the hbase files.
so it sounds like the migrate is in order.
-Dmitriy
On Tue, Sep 28, 2010 at 9:15 AM, Stack <st...@duboce.net> wrote:
> On Mon, Sep 27, 2010 at 7:52 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> > Hi,
> >
> > i would be very grateful if somebody could clarify the following for me
> > please. (0.20.5)
> >
> > yesterday we lost a short table (~100 rows) in production without a
> trace.
> > no matter how deep i looked in the logs of regionservers and the master,
> i
> > haven't got a clue how it might have happened.
> >
> > When i looked at the table though, i did not find any files, which may
> mean
> > that it never got flushed from WAL and got compacted. When i
> reconstituted
> > it and ran compaction, the file did finally appear .
> >
>
> What did you do to 'reconstitute'?
>
> FYI, edits go first to WAL and then to memstore. A file will not
> appear in the filesystem until memstore flushes. 100 rows is probably
> not enough to bring on a flush.
>
>
> > also, in the .log (hlog) directory, i noticed some duplicate entries with
> > both long and short host names (i.e. something like 'data4,...' and
> > 'data4.foo.bar,...' ) which may result from the moment we decided to
> switch
> > to /etc/hosts name resolution instead of dns (just to see if that'll
> improve
> > our networking issues).
> >
>
> Yeah, probably.
>
> Our hostname lookup is done once in 0.90.x and forever after we keep
> on w/ that name regardless so this should not happen going forward.
>
>
> > So i have the following questions :
> >
> > 1 -- when is WAL (or hlog, if it's the same?) is triggered and the actual
> > tablet file is built? is it a size-based threshold? is there an max age
> > threshold?
>
> Yes, the hbase WAL is also referred to as hlog and our WAL is
> implemented by the o.a.h.h.regionserver.wal.HLog class.
>
> Every edit first is appended to the WAL that each regionserver keeps up.
>
> But, in 0.20 hbase, our WAL is mostly ineffective given as there is no
> append support in hadoop 0.20 hdfs; basically only if the file is
> successfully closed will edits be preserved (This state has changed in
> 0.89.x in that the WAL append now works Make sure you are running
> with the hadoop 0.20-append branch or CDH3b2 to ensure your 0.89.x
> install WAL works properly).
>
>
> > 2 -- if the region server crashes, its hlog is supposed to be split and
> > recovered, right?
>
> Yes
>
>
> is there a situation when hlog can be lost? I suppose it
> > doesn't matter that the region server with the same name never goes
> online
> > again (e.g. if it starts using short name instead of FQDN)?
> >
>
> I'd have to check the code but I think the messing w/ hostnames would
> not be a reason to lose edits. I think we go ahead and split the logs
> for files even if they are not associated with a particular host; i.e.
> though you "changed" hostnames, and our WALs are associated with
> hosts, we pick up any straggler log files anyways. Check your master
> logs to confirm.
>
> St.Ack
>
Re: WAL, failover questions
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
St. Ack,
just to make sure: by cdh3b2 aka 'hadoop-append' you imply hadoop 0.20.2+320
from cdh3 distro, right?
Thank you.
-Dmitriy
On Tue, Sep 28, 2010 at 9:15 AM, Stack <st...@duboce.net> wrote:
> On Mon, Sep 27, 2010 at 7:52 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> > Hi,
> >
> > i would be very grateful if somebody could clarify the following for me
> > please. (0.20.5)
> >
> > yesterday we lost a short table (~100 rows) in production without a
> trace.
> > no matter how deep i looked in the logs of regionservers and the master,
> i
> > haven't got a clue how it might have happened.
> >
> > When i looked at the table though, i did not find any files, which may
> mean
> > that it never got flushed from WAL and got compacted. When i
> reconstituted
> > it and ran compaction, the file did finally appear .
> >
>
> What did you do to 'reconstitute'?
>
> FYI, edits go first to WAL and then to memstore. A file will not
> appear in the filesystem until memstore flushes. 100 rows is probably
> not enough to bring on a flush.
>
>
> > also, in the .log (hlog) directory, i noticed some duplicate entries with
> > both long and short host names (i.e. something like 'data4,...' and
> > 'data4.foo.bar,...' ) which may result from the moment we decided to
> switch
> > to /etc/hosts name resolution instead of dns (just to see if that'll
> improve
> > our networking issues).
> >
>
> Yeah, probably.
>
> Our hostname lookup is done once in 0.90.x and forever after we keep
> on w/ that name regardless so this should not happen going forward.
>
>
> > So i have the following questions :
> >
> > 1 -- when is WAL (or hlog, if it's the same?) is triggered and the actual
> > tablet file is built? is it a size-based threshold? is there an max age
> > threshold?
>
> Yes, the hbase WAL is also referred to as hlog and our WAL is
> implemented by the o.a.h.h.regionserver.wal.HLog class.
>
> Every edit first is appended to the WAL that each regionserver keeps up.
>
> But, in 0.20 hbase, our WAL is mostly ineffective given as there is no
> append support in hadoop 0.20 hdfs; basically only if the file is
> successfully closed will edits be preserved (This state has changed in
> 0.89.x in that the WAL append now works Make sure you are running
> with the hadoop 0.20-append branch or CDH3b2 to ensure your 0.89.x
> install WAL works properly).
>
>
> > 2 -- if the region server crashes, its hlog is supposed to be split and
> > recovered, right?
>
> Yes
>
>
> is there a situation when hlog can be lost? I suppose it
> > doesn't matter that the region server with the same name never goes
> online
> > again (e.g. if it starts using short name instead of FQDN)?
> >
>
> I'd have to check the code but I think the messing w/ hostnames would
> not be a reason to lose edits. I think we go ahead and split the logs
> for files even if they are not associated with a particular host; i.e.
> though you "changed" hostnames, and our WALs are associated with
> hosts, we pick up any straggler log files anyways. Check your master
> logs to confirm.
>
> St.Ack
>
Re: WAL, failover questions
Posted by Stack <st...@duboce.net>.
On Mon, Sep 27, 2010 at 7:52 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Hi,
>
> i would be very grateful if somebody could clarify the following for me
> please. (0.20.5)
>
> yesterday we lost a short table (~100 rows) in production without a trace.
> no matter how deep i looked in the logs of regionservers and the master, i
> haven't got a clue how it might have happened.
>
> When i looked at the table though, i did not find any files, which may mean
> that it never got flushed from WAL and got compacted. When i reconstituted
> it and ran compaction, the file did finally appear .
>
What did you do to 'reconstitute'?
FYI, edits go first to WAL and then to memstore. A file will not
appear in the filesystem until memstore flushes. 100 rows is probably
not enough to bring on a flush.
> also, in the .log (hlog) directory, i noticed some duplicate entries with
> both long and short host names (i.e. something like 'data4,...' and
> 'data4.foo.bar,...' ) which may result from the moment we decided to switch
> to /etc/hosts name resolution instead of dns (just to see if that'll improve
> our networking issues).
>
Yeah, probably.
Our hostname lookup is done once in 0.90.x and forever after we keep
on w/ that name regardless so this should not happen going forward.
> So i have the following questions :
>
> 1 -- when is WAL (or hlog, if it's the same?) is triggered and the actual
> tablet file is built? is it a size-based threshold? is there an max age
> threshold?
Yes, the hbase WAL is also referred to as hlog and our WAL is
implemented by the o.a.h.h.regionserver.wal.HLog class.
Every edit first is appended to the WAL that each regionserver keeps up.
But, in 0.20 hbase, our WAL is mostly ineffective given as there is no
append support in hadoop 0.20 hdfs; basically only if the file is
successfully closed will edits be preserved (This state has changed in
0.89.x in that the WAL append now works Make sure you are running
with the hadoop 0.20-append branch or CDH3b2 to ensure your 0.89.x
install WAL works properly).
> 2 -- if the region server crashes, its hlog is supposed to be split and
> recovered, right?
Yes
is there a situation when hlog can be lost? I suppose it
> doesn't matter that the region server with the same name never goes online
> again (e.g. if it starts using short name instead of FQDN)?
>
I'd have to check the code but I think the messing w/ hostnames would
not be a reason to lose edits. I think we go ahead and split the logs
for files even if they are not associated with a particular host; i.e.
though you "changed" hostnames, and our WALs are associated with
hosts, we pick up any straggler log files anyways. Check your master
logs to confirm.
St.Ack