[jira] Updated: (HBASE-1416) Pool of commit loggers in each HRegionServer

["Andrew Purtell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-1416:
----------------------------------

    Fix Version/s: 0.21.0

Up on hbase-dev@ Ryan writes:
{quote}
we need to make hlog flush faster, it currently does only 700 ops/sec
when we flush every entry.

it'd be nice if we could do something clever, such as:

- use multiple logs
- detect multiple waiting clients and better batch their commits
- group commits for bulk import
{quote}

This issue addresses the first point. 

While considering this, dynamically size the pool according to a concurrency measure. Spin up new writers on demand until some configurable upper bound. A simple strategy to try first might be 2 * ceil(log(load)), smoothed. Terminate excess writers at roll time to hold down unnecessary HDFS resource use.

In HLog.doWrite we write each HLogKey and KeyValue to the log, which is a SequenceFile. Use hfile instead? Can HFile do I/O batching? Otherwise I think to group commits we'd need to introduce a new writable which bundles edits together. 

Moving into 0.21.

> Pool of commit loggers in each HRegionServer
> --------------------------------------------
>
>                 Key: HBASE-1416
>                 URL: https://issues.apache.org/jira/browse/HBASE-1416
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.21.0
>
>
> HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction.   This issue is about implementing the pool of writers.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.