[jira] [Commented] (HDFS-13189) Standby NameNode should roll active edit log when checkpointing

["xuzq (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830014#comment-16830014 ] 

xuzq commented on HDFS-13189:
-----------------------------

I think the edit log roll and tailing edits need to do in separate threads in Standby, because they all independency. 

 

There is one other  problem to solve. In current trunk, both Observers and Standbys  can rollEdit, it may produce too many small segments. And will produce some negative effects, such as too many small files in each JournalNode. 

> Standby NameNode should roll active edit log when checkpointing
> ---------------------------------------------------------------
>
>                 Key: HDFS-13189
>                 URL: https://issues.apache.org/jira/browse/HDFS-13189
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Chao Sun
>            Priority: Minor
>
> When the SBN is doing checkpointing, it will hold the {{cpLock}}. In the current implementation of edit log tailer thread, it will first check and roll active edit log, and then tail and apply edits. In the case of checkpointing, it will be blocked on the {{cpLock}} and will not roll the edit log.
> It seems there is no dependency between the edit log roll and tailing edits, so a better may be to do these in separate threads. This will be helpful for people who uses the observer feature without in-progress edit log tailing. 
> An alternative is to configure {{dfs.namenode.edit.log.autoroll.multiplier.threshold}} and {{dfs.namenode.edit.log.autoroll.check.interval.ms}} to let ANN roll its own log more frequently in case SBN is stuck on the lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org