[jira] [Commented] (ACCUMULO-1083) add concurrency to HDFS write-ahead log

["Eric Newton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584219#comment-13584219 ] 

Eric Newton commented on ACCUMULO-1083:
---------------------------------------

Concurrent, as in, a sub-set of tablets use a log, or concurrent, as in, there are multiple log files available to log to, as described in the Big Table paper?

Note that in 1.5, default replication of the WAL is probably 3, whereas it used to be 2.

                
> add concurrency to HDFS write-ahead log
> ---------------------------------------
>
>                 Key: ACCUMULO-1083
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1083
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: logger, tserver
>            Reporter: Adam Fuchs
>
> When running tablet servers on beefy nodes (lots of disks), the write-ahead log can be a serious bottleneck. Today we ran a test of 1.5-SNAPSHOT on an 8-node (plus a master node) cluster in which the nodes had 32 cores and 15 drives each. Running with write-ahead log off resulted in a >4x performance improvement sustained over a long period.
> I believe the culprit is that the WAL is only using one file at a time per tablet server, which means HDFS is only appending to one drive (plus replicas). If we increase the number of concurrent WAL files supported on a tablet server we could probably drastically improve the performance on systems with many disks. As it stands, I believe Accumulo is significantly more optimized for a larger number of smaller nodes (3-4 drives).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira