Region server instability with large number of deletes

[Mark Greene <mg...@hubspot.com>]

Hi All,

I'm running HBase 0.92.1-cdh4.1.2.

I'm seeing a region server get into a state where they constantly emit
LeaseExceptions and ClosedChannelException from the IPC handler under load
that is several standard deviations above what we normally see.

One thing that's different about this cluster is we do a lot of deletes
against it. Ultimately all the data in one of our tables is deleted.

When the region server was in this state, the application was doing a lot
of scans, single gets, and then a series of batch deletes. While this cycle
was churning at a higher than average rate then what I normally see, it is
being done in a single thread. There are of course other processes reading
and writing to the cluster while this was going on but the observed load
from those were nothing out of the norm.

So I'm wondering if the large number of deletes via batch is a usage
profile that isn't optimized for hbase?

Other info:
-We just migrated to cdh4.1.2 a few days ago
-Currently do not have major compactions running
-Saw the same behavior in cdh3
-CPU and iowait were low during the time I observed these errors happening.

Thanks in advance,
Mark

Re: Region server instability with large number of deletes

[lars hofhansl <lh...@yahoo.com>]

That is fixed in 0.94 (can't find the exact issue now, though)

Before the fix in 0.94 the region server would skip deleted columns one by one in order to advance to the next row.
Now it seeks ahead, instead.

In 0.92 I think your only option is to follow your batch delete with a major compaction.


-- Lars



________________________________
 From: Mark Greene <mg...@hubspot.com>
To: user@hbase.apache.org 
Sent: Wednesday, December 12, 2012 11:16 AM
Subject: Region server instability with large number of deletes
 
Hi All,

I'm running HBase 0.92.1-cdh4.1.2.

I'm seeing a region server get into a state where they constantly emit
LeaseExceptions and ClosedChannelException from the IPC handler under load
that is several standard deviations above what we normally see.

One thing that's different about this cluster is we do a lot of deletes
against it. Ultimately all the data in one of our tables is deleted.

When the region server was in this state, the application was doing a lot
of scans, single gets, and then a series of batch deletes. While this cycle
was churning at a higher than average rate then what I normally see, it is
being done in a single thread. There are of course other processes reading
and writing to the cluster while this was going on but the observed load
from those were nothing out of the norm.

So I'm wondering if the large number of deletes via batch is a usage
profile that isn't optimized for hbase?

Other info:
-We just migrated to cdh4.1.2 a few days ago
-Currently do not have major compactions running
-Saw the same behavior in cdh3
-CPU and iowait were low during the time I observed these errors happening.

Thanks in advance,
Mark