[jira] [Comment Edited] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only

["Arpit Agarwal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295639#comment-16295639 ] 

Arpit Agarwal edited comment on HBASE-14790 at 12/18/17 8:38 PM:
-----------------------------------------------------------------

This looks interesting.

[~Apache9], any interest in contributing this to HDFS? Also do you have any measurements for the latency benefit? I skimmed the comments but didn't see it so I apologize if it's there already. 

(Thanks for the heads up [~yuzhihong@gmail.com]).


was (Author: arpitagarwal):
This looks interesting.

[~Apache9], any interest in contributing this to HDFS? Also do you have any measurements for the latency benefit? I skimmed the comments but didn't see it so I apologize if it's there already. 

> Implement a new DFSOutputStream for logging WAL only
> ----------------------------------------------------
>
>                 Key: HBASE-14790
>                 URL: https://issues.apache.org/jira/browse/HBASE-14790
>             Project: HBase
>          Issue Type: Improvement
>          Components: wal
>            Reporter: Duo Zhang
>            Assignee: Duo Zhang
>             Fix For: 2.0.0-beta-1
>
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all purposes. But in fact, we do not need most of the features if we only want to log WAL. For example, we do not need pipeline recovery since we could just close the old logger and open a new one. And also, we do not need to write multiple blocks since we could also open a new logger if the old file is too large.
> And the most important thing is that, it is hard to handle all the corner cases to avoid data loss or data inconsistency(such as HBASE-14004) when using original DFSOutputStream due to its complicated logic. And the complicated logic also force us to use some magical tricks to increase performance. For example, we need to use multiple threads to call {{hflush}} when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when logging WAL. For correctness, and also for performance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)