multi-threaded HTablePool, incrementColumnValue, compaction and large data set

[Neil Yalowitz <ne...@gmail.com>]

I'm seeing something unusual here and I wanted to see if it has occurred
for any other HBase 0.90 users.  I've read several emails here that
recommend NOT using multi-threading in an MR job, so that's certainly under
consideration.  If anyone could add to their experiences with
multi-threading in an MR job it would be very helpful.  We are testing both
implementations (with threading and without), but the threaded solution is
causing the problem.

We are processing log files with PUTs in the Map and a followup
incrementColumnValue() to a separate "counts" table in the Reducer.  The
reduce phase uses multi-threading.  The Reducer initializes an HTablePool
in the setup(), starts threads in the reduce() (to a
Java BlockingQueue/CompletionService) which do the incrementColumnValue()
and depending on the value returned create a PUT in the "counter" table,
and in the cleanup() performs a completionService.take() which is ignored
and flushes the PUTs queued by the threads.

There are no issues for approximately the first 100GB of data inserted.
 After approximately 100GB however, every subsequent job has a freeze
during the Reduce phase.  What I see happening is at some point the Reduce
(where the incrementColumnValue() takes place) tasks are "hung" and
eventually killed with reason: task client has not responded for 600
seconds.  The counters in the reduce job seem to grow briefly but then all
the tasks' counter stop increasing and the task is eventually killed.

Oddly, the problem does not occur if compaction is completely disabled (not
just major, but also setting hbase.hstore.compactionThreshold = 9999999
and hbase.hstore.blockingStoreFiles = 9999999).

Could there be a bug with HTablePool for large datasets and compaction?
 Again, this works as expected for approximately the first 100 jobs (1GB
each) but consistently fails after that.  Also to repeat, the problem does
not occur with ALL compaction disabled.

Difficult problem to describe, but I'm hoping someone may have some
feedback and/or similar experiences.  I can provide code examples if anyone
is curious.



Neil Yalowitz

Re: multi-threaded HTablePool, incrementColumnValue, compaction and large data set

[Sambit Tripathy <sa...@gmail.com>]

So did you get any success with the problem? Well, I think you can try
using it with Asynchbase, a hbase client used in OpenTSDB.



On Mon, Jan 16, 2012 at 6:46 AM, Neil Yalowitz <ne...@gmail.com>wrote:

> I'm seeing something unusual here and I wanted to see if it has occurred
> for any other HBase 0.90 users.  I've read several emails here that
> recommend NOT using multi-threading in an MR job, so that's certainly under
> consideration.  If anyone could add to their experiences with
> multi-threading in an MR job it would be very helpful.  We are testing both
> implementations (with threading and without), but the threaded solution is
> causing the problem.
>
> We are processing log files with PUTs in the Map and a followup
> incrementColumnValue() to a separate "counts" table in the Reducer.  The
> reduce phase uses multi-threading.  The Reducer initializes an HTablePool
> in the setup(), starts threads in the reduce() (to a
> Java BlockingQueue/CompletionService) which do the incrementColumnValue()
> and depending on the value returned create a PUT in the "counter" table,
> and in the cleanup() performs a completionService.take() which is ignored
> and flushes the PUTs queued by the threads.
>
> There are no issues for approximately the first 100GB of data inserted.
>  After approximately 100GB however, every subsequent job has a freeze
> during the Reduce phase.  What I see happening is at some point the Reduce
> (where the incrementColumnValue() takes place) tasks are "hung" and
> eventually killed with reason: task client has not responded for 600
> seconds.  The counters in the reduce job seem to grow briefly but then all
> the tasks' counter stop increasing and the task is eventually killed.
>
> Oddly, the problem does not occur if compaction is completely disabled (not
> just major, but also setting hbase.hstore.compactionThreshold = 9999999
> and hbase.hstore.blockingStoreFiles = 9999999).
>
> Could there be a bug with HTablePool for large datasets and compaction?
>  Again, this works as expected for approximately the first 100 jobs (1GB
> each) but consistently fails after that.  Also to repeat, the problem does
> not occur with ALL compaction disabled.
>
> Difficult problem to describe, but I'm hoping someone may have some
> feedback and/or similar experiences.  I can provide code examples if anyone
> is curious.
>
>
>
> Neil Yalowitz
>