hadoop 0.21 hdfs sync and hbase

[stack <st...@duboce.net>]

HBase TRUNK has recent versions of hadoop hdfs jars committed to its lib
dir.  Our WAL (HLog) has been amended so we call through to the new hadoop
sync (hflush).  A basic unit test that exercises the new sync has been added
where a Reader can read all edits up to the last sync and its passing.  Next
up is testing out on a (hadoop 0.21) cluster to see well it works under
load.  I would suggest that all who can pitch in with the testing of this
critical feature.  Raise any issues found either here or over in hdfs.  Be
aware of hdfs-630 when testing -- you'll probably need to apply it to your
hadoop install.

Oh, TRUNK should be back to some normality now though hopefully, its going
to undergo much radical distress between here and our 0.21 release.
Good stuff,
St.Ack

Re: hadoop 0.21 hdfs sync and hbase

[Ryan Rawson <ry...@gmail.com>]

Be warned - sync makes things slowwwwwwwwwwwww. Really slow. Expect 1/50th
the perf you saw.

I have some ideas, but for now relax a bit when you see 1000 ops.sec :-)

On Oct 9, 2009 5:23 PM, "stack" <st...@duboce.net> wrote:

HBase TRUNK has recent versions of hadoop hdfs jars committed to its lib
dir.  Our WAL (HLog) has been amended so we call through to the new hadoop
sync (hflush).  A basic unit test that exercises the new sync has been added
where a Reader can read all edits up to the last sync and its passing.  Next
up is testing out on a (hadoop 0.21) cluster to see well it works under
load.  I would suggest that all who can pitch in with the testing of this
critical feature.  Raise any issues found either here or over in hdfs.  Be
aware of hdfs-630 when testing -- you'll probably need to apply it to your
hadoop install.

Oh, TRUNK should be back to some normality now though hopefully, its going
to undergo much radical distress between here and our 0.21 release.
Good stuff,
St.Ack