You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/05/13 07:32:45 UTC
[jira] Created: (HBASE-1416) Pool of commit loggers in each
HRegionServer
Pool of commit loggers in each HRegionServer
--------------------------------------------
Key: HBASE-1416
URL: https://issues.apache.org/jira/browse/HBASE-1416
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction. This issue is about implementing the pool of writers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HBASE-1416) Pool of commit loggers in each
HRegionServer
Posted by Ryan Rawson <ry...@gmail.com>.
With deferred log flush we can choose between high durability and speed on a
per table basis. I think we should shelf this for now, since with write
buffering in the client and the adjusted sync calls, we are very fast at
bulk data import.
On Dec 19, 2009 1:47 PM, "Andrew Purtell (JIRA)" <ji...@apache.org> wrote:
[
https://issues.apache.org/jira/browse/HBASE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792910#action_12792910]
Andrew Purtell commented on HBASE-1416:
---------------------------------------
Do we still want to do this? Seems some of Ryan's ideas have been
implemented as HBASE-1939 already.
I was playing around with this the other day after committing HBASE-2059. On
that note, implementing an HLog.Writer that stripes edits over a pool of
logs under a directory instead of a single file works ok on read and write
paths but split and roll code assumes the log is a single file so barf when
they access a dir instead; the HLog.Writer interface would need to be
widened with split and roll methods to make that work cleanly. Anyway, sync
is still a log pole and with striped writes over a pool of HLogs there are
more dirty blocks to sync. Unless the current edit load is high we end up
dirtying small bits of a bunch of blocks where instead we should just write
them all into one as they'll all fit in there.
> Pool of commit loggers in each HRegionServer >
-------------------------------------------- > > ...
[jira] Commented: (HBASE-1416) Pool of commit loggers in each
HRegionServer
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770730#action_12770730 ]
Andrew Purtell commented on HBASE-1416:
---------------------------------------
Split out writes into a commit thread per WAL after HBASE-1939.
> Pool of commit loggers in each HRegionServer
> --------------------------------------------
>
> Key: HBASE-1416
> URL: https://issues.apache.org/jira/browse/HBASE-1416
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.21.0
>
>
> HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction. This issue is about implementing the pool of writers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1416) Pool of commit loggers in each
HRegionServer
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792910#action_12792910 ]
Andrew Purtell commented on HBASE-1416:
---------------------------------------
Do we still want to do this? Seems some of Ryan's ideas have been implemented as HBASE-1939 already.
I was playing around with this the other day after committing HBASE-2059. On that note, implementing an HLog.Writer that stripes edits over a pool of logs under a directory instead of a single file works ok on read and write paths but split and roll code assumes the log is a single file so barf when they access a dir instead; the HLog.Writer interface would need to be widened with split and roll methods to make that work cleanly. Anyway, sync is still a log pole and with striped writes over a pool of HLogs there are more dirty blocks to sync. Unless the current edit load is high we end up dirtying small bits of a bunch of blocks where instead we should just write them all into one as they'll all fit in there.
> Pool of commit loggers in each HRegionServer
> --------------------------------------------
>
> Key: HBASE-1416
> URL: https://issues.apache.org/jira/browse/HBASE-1416
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.21.0
>
>
> HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction. This issue is about implementing the pool of writers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1416) Pool of commit loggers in each
HRegionServer
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-1416.
-----------------------------------
Resolution: Later
Resolving as Later. I believe everyone agrees this issue is something to defer, probably indefinitely.
> Pool of commit loggers in each HRegionServer
> --------------------------------------------
>
> Key: HBASE-1416
> URL: https://issues.apache.org/jira/browse/HBASE-1416
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Minor
>
> HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction. This issue is about implementing the pool of writers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1416) Pool of commit loggers in each
HRegionServer
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829446#action_12829446 ]
Jean-Daniel Cryans commented on HBASE-1416:
-------------------------------------------
I agree, since we always write on local node first then having multiple writers won't help. Also the 2 latter points were implemented so the appending/hflushing situation is actually pretty good.
> Pool of commit loggers in each HRegionServer
> --------------------------------------------
>
> Key: HBASE-1416
> URL: https://issues.apache.org/jira/browse/HBASE-1416
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Minor
>
> HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction. This issue is about implementing the pool of writers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1416) Pool of commit loggers in each
HRegionServer
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792924#action_12792924 ]
stack commented on HBASE-1416:
------------------------------
Below are Ryan's comments from the list.
"With deferred log flush we can choose between high durability and speed on a
per table basis. I think we should shelf this for now, since with write
buffering in the client and the adjusted sync calls, we are very fast at
bulk data import."
J-D says write speeds are up with hbase-1939.
I'm good w/ punting out to 0.22 or beyond. Let me move it.
> Pool of commit loggers in each HRegionServer
> --------------------------------------------
>
> Key: HBASE-1416
> URL: https://issues.apache.org/jira/browse/HBASE-1416
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.21.0
>
>
> HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction. This issue is about implementing the pool of writers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1416) Pool of commit loggers in each
HRegionServer
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1416:
-------------------------
Priority: Minor (was: Major)
Fix Version/s: (was: 0.21.0)
Moved it out of 0.21 and made it trivial priority.
> Pool of commit loggers in each HRegionServer
> --------------------------------------------
>
> Key: HBASE-1416
> URL: https://issues.apache.org/jira/browse/HBASE-1416
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Minor
>
> HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction. This issue is about implementing the pool of writers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1416) Pool of commit loggers in each
HRegionServer
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1416:
----------------------------------
Fix Version/s: 0.21.0
Up on hbase-dev@ Ryan writes:
{quote}
we need to make hlog flush faster, it currently does only 700 ops/sec
when we flush every entry.
it'd be nice if we could do something clever, such as:
- use multiple logs
- detect multiple waiting clients and better batch their commits
- group commits for bulk import
{quote}
This issue addresses the first point.
While considering this, dynamically size the pool according to a concurrency measure. Spin up new writers on demand until some configurable upper bound. A simple strategy to try first might be 2 * ceil(log(load)), smoothed. Terminate excess writers at roll time to hold down unnecessary HDFS resource use.
In HLog.doWrite we write each HLogKey and KeyValue to the log, which is a SequenceFile. Use hfile instead? Can HFile do I/O batching? Otherwise I think to group commits we'd need to introduce a new writable which bundles edits together.
Moving into 0.21.
> Pool of commit loggers in each HRegionServer
> --------------------------------------------
>
> Key: HBASE-1416
> URL: https://issues.apache.org/jira/browse/HBASE-1416
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.21.0
>
>
> HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction. This issue is about implementing the pool of writers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.