.logs directory keeps growing

[Alok Singh <al...@urbanairship.com>]

Environment
Hbase: 0.92
Hadoop: hadoop-0.20.2-cdh3u3

I am testing hbase 0.92 for a new storage system we are building. In
the tests, I insert around 2-3 billion rows and then run some
scans/queries against it to test the performance. Once the tests are
complete, I drop all of the tables and recreate them to clear out the
test data. But, it doesn't look like the /hbase/root/.logs is getting
cleared. It seems to just keep growing, and the only way to clear it
out is to restart the hbase cluster. After the restarts, the
regionserver logs are filled with this:
---------
12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory
doesn't exist: hdfs://10.130.1.15:7080/foo/hbase/root/device/9605bdd29d02212f23499296542341d8.
It is very likely that it was already split so it's safe to discard
those edits.
12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory
doesn't exist: hdfs://10.130.1.15:7080/foo/hbase/root/device.device/b3b96cd17e683e52f8c72e14229f5567.
It is very likely that it was already split so it's safe to discard
those edits.
12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory
doesn't exist: hdfs://10.130.1.15:7080/foo/hbase/root/device/83832bcdc8a1bde1094d9039594cb067.
It is very likely that it was already split so it's safe to discard
those edits.
12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory
doesn't exist: hdfs://10.130.1.15:7080/foo/hbase/root/device/140ded7a67c8cf2a0e7cbbf037033889.
It is very likely that it was already split so it's safe to discard
those edits.
.....
----------

Size of /hbase/root/device: 110GB
Size of /hbase/root/.logs: 770GB

Alok

Re: .logs directory keeps growing

[Jean-Daniel Cryans <jd...@apache.org>]

Many things going on here.

1- The logs are receiving edits from all the regions, so a log can't
be cleared until all the regions it contains are flushed.

2- When you close regions (drop a tables implies doing that), it
doesn't force roll logs (it would be very bad on performance) so they
won't get cleared right away even if they can be.

3- Those HLogSplitter lines seem to indicate that you didn't shut down
HBase properly since it had to replay the logs.

Hope this helps,

J-D

On Wed, Feb 22, 2012 at 3:03 PM, Alok Singh <al...@urbanairship.com> wrote:
> Environment
> Hbase: 0.92
> Hadoop: hadoop-0.20.2-cdh3u3
>
> I am testing hbase 0.92 for a new storage system we are building. In
> the tests, I insert around 2-3 billion rows and then run some
> scans/queries against it to test the performance. Once the tests are
> complete, I drop all of the tables and recreate them to clear out the
> test data. But, it doesn't look like the /hbase/root/.logs is getting
> cleared. It seems to just keep growing, and the only way to clear it
> out is to restart the hbase cluster. After the restarts, the
> regionserver logs are filled with this:
> ---------
> 12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory
> doesn't exist: hdfs://10.130.1.15:7080/foo/hbase/root/device/9605bdd29d02212f23499296542341d8.
> It is very likely that it was already split so it's safe to discard
> those edits.
> 12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory
> doesn't exist: hdfs://10.130.1.15:7080/foo/hbase/root/device.device/b3b96cd17e683e52f8c72e14229f5567.
> It is very likely that it was already split so it's safe to discard
> those edits.
> 12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory
> doesn't exist: hdfs://10.130.1.15:7080/foo/hbase/root/device/83832bcdc8a1bde1094d9039594cb067.
> It is very likely that it was already split so it's safe to discard
> those edits.
> 12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory
> doesn't exist: hdfs://10.130.1.15:7080/foo/hbase/root/device/140ded7a67c8cf2a0e7cbbf037033889.
> It is very likely that it was already split so it's safe to discard
> those edits.
> .....
> ----------
>
> Size of /hbase/root/device: 110GB
> Size of /hbase/root/.logs: 770GB
>
> Alok