[jira] [Resolved] (ACCUMULO-1083) add concurrency to HDFS write-ahead log

["Josh Elser (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ACCUMULO-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Elser resolved ACCUMULO-1083.
----------------------------------
    Resolution: Won't Fix

Rather stale and potentially irrelevant after ACCUMULO-3423. Can reopen if it someone wants to take this up.

> add concurrency to HDFS write-ahead log
> ---------------------------------------
>
>                 Key: ACCUMULO-1083
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1083
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: tserver
>            Reporter: Adam Fuchs
>         Attachments: walog-performance.jpg, walog-replication-factor-performance.jpg
>
>
> When running tablet servers on beefy nodes (lots of disks), the write-ahead log can be a serious bottleneck. Today we ran a continuous ingest test of 1.5-SNAPSHOT on an 8-node (plus a master node) cluster in which the nodes had 32 cores and 15 drives each. Running with write-ahead log off resulted in a >4x performance improvement sustained over a long period.
> I believe the culprit is that the WAL is only using one file at a time per tablet server, which means HDFS is only appending to one drive (plus replicas). If we increase the number of concurrent WAL files supported on a tablet server we could probably drastically improve the performance on systems with many disks. As it stands, I believe Accumulo is significantly more optimized for a larger number of smaller nodes (3-4 drives).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)