[jira] [Comment Edited] (HBASE-6116) Allow parallel HDFS writes for HLogs.

["Andrew Purtell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400075#comment-13400075 ] 

Andrew Purtell edited comment on HBASE-6116 at 6/23/12 11:39 PM:
-----------------------------------------------------------------

Attached is a comparison of pipelined vs. parallel sync differences on two identical (but different) 5 slave EC2 clusters. I modified HBase to use a new histogram metric for recording HLog sync latency and then ran a write dominant workload on each cluster using LoadTestTool for 60 minutes and captured RegionServer metrics at one second intervals.

The first tab of the spreadsheet describes the experiment parameters. The second shows mean, 99th percentile, and standard deviation for pipelined syncs as reported. The third shows mean, 99th percentile, and standard deviation for parallel syncs as reported. The fourth has some simple graphs I threw together for illustration. The remaining tabs contain the detail of the captured metrics for each host.

Edit: Note all metrics are in milliseconds.
                
      was (Author: apurtell):
    Attached is a comparison of pipelined vs. parallel sync differences on two identical (but different) 5 slave EC2 clusters. I modified HBase to use a new histogram metric for recording HLog sync latency and then ran a write dominant workload on each cluster using LoadTestTool for 60 minutes and captured RegionServer metrics at one second intervals.

The first tab of the spreadsheet describes the experiment parameters. The second shows mean, 99th percentile, and standard deviation for pipelined syncs as reported. The third shows mean, 99th percentile, and standard deviation for parallel syncs as reported. The fourth has some simple graphs I threw together for illustration. The remaining tabs contain the detail of the captured metrics for each host.
                  
> Allow parallel HDFS writes for HLogs.
> -------------------------------------
>
>                 Key: HBASE-6116
>                 URL: https://issues.apache.org/jira/browse/HBASE-6116
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Lars Hofhansl
>         Attachments: 6116-v1.txt, pipelined-vs-parallel-comparison.zip
>
>
> In HDFS-1783 I adapted Dhrubas changes to be used in Hadoop trunk.
> This issue will include the necessary reflection changes to optionally enable this for the WALs in HBase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira