Blocking of callers during compaction

[Mathias Herberts <ma...@gmail.com>]

I've encountered a situation lately where lots of regions need
compaction at the same time. This leads to callers being blocked
waiting to grab the monitor to enter 'reclaimMemStoreMemory'.

When the number of regions is large, the blocking can last a long time.

In my specific case, the callers are Reducer instances and the timeout
of the reducer (mapred.task.timeout) actually gets reached and the
Reducer gets restarted.

I was thinking about a possible enhancement where we could specify in
HTable that in case of compaction we want to be pushed back
immediately instead of being blocked.

This could be done by adding a boolean to Put objects that would be
set in HTable.put(), reflecting the HTable setting (thus avoiding to
have several values of this flag in a list of Put). The modification
of this flag would trigger a flushCommits so all Put instances in the
buffer carry the same value of the flag.

The behavior in case a request is pushed back would be to throw an
IOException, leaving the write buffer as it was prior to the call.

This would allow in my case the Reducer to wait on its side, modifying
its status so mapred.task.timeout is not reahed.

What do you think?

Mathias.

Re: Blocking of callers during compaction

[Mathias Herberts <ma...@gmail.com>]

On Tue, Aug 18, 2009 at 16:33, Jean-Daniel Cryans<jd...@apache.org> wrote:
> Mathias,
>
> There is indeed something to do. At least, we should not clear the
> buffer if the put fails for some reason.

That would be a better behavior yes.

> WRT to the blocking, last night Stack and Jon fixed an issue where we
> were blocking too much (HBASE-1771). I guess that's what you hit since
> reclaimMemStoreMemory will flush regions when it's able to do so. So
> updating to RC2 should help your case. Also, if you have enough RAM,
> you could change hbase.regionserver.global.memstore.upperLimit and
> lowerLimit to something higher since 40% of 1GB is very different than
> 40% of 4GB. Then you can also set a higher
> hbase.hstore.blockingStoreFiles.

I'll give RC2 a try, thanks for the hint.

Mathias.

Re: Blocking of callers during compaction

[Jean-Daniel Cryans <jd...@apache.org>]

Mathias,

There is indeed something to do. At least, we should not clear the
buffer if the put fails for some reason.

WRT to the blocking, last night Stack and Jon fixed an issue where we
were blocking too much (HBASE-1771). I guess that's what you hit since
reclaimMemStoreMemory will flush regions when it's able to do so. So
updating to RC2 should help your case. Also, if you have enough RAM,
you could change hbase.regionserver.global.memstore.upperLimit and
lowerLimit to something higher since 40% of 1GB is very different than
40% of 4GB. Then you can also set a higher
hbase.hstore.blockingStoreFiles.

J-D

On Tue, Aug 18, 2009 at 10:13 AM, Mathias
Herberts<ma...@gmail.com> wrote:
> I've encountered a situation lately where lots of regions need
> compaction at the same time. This leads to callers being blocked
> waiting to grab the monitor to enter 'reclaimMemStoreMemory'.
>
> When the number of regions is large, the blocking can last a long time.
>
> In my specific case, the callers are Reducer instances and the timeout
> of the reducer (mapred.task.timeout) actually gets reached and the
> Reducer gets restarted.
>
> I was thinking about a possible enhancement where we could specify in
> HTable that in case of compaction we want to be pushed back
> immediately instead of being blocked.
>
> This could be done by adding a boolean to Put objects that would be
> set in HTable.put(), reflecting the HTable setting (thus avoiding to
> have several values of this flag in a list of Put). The modification
> of this flag would trigger a flushCommits so all Put instances in the
> buffer carry the same value of the flag.
>
> The behavior in case a request is pushed back would be to throw an
> IOException, leaving the write buffer as it was prior to the call.
>
> This would allow in my case the Reducer to wait on its side, modifying
> its status so mapred.task.timeout is not reahed.
>
> What do you think?
>
> Mathias.
>