You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "JiangHua Zhu (Jira)" <ji...@apache.org> on 2021/04/09 03:41:00 UTC
[jira] [Commented] (HDFS-15887) Make LogRoll and TailEdits execute
in parallel
[ https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17317614#comment-17317614 ]
JiangHua Zhu commented on HDFS-15887:
-------------------------------------
[~weichiu] [~hexiaoqiao], I submitted some code. Can you give me a review?
Thank you very much.
> Make LogRoll and TailEdits execute in parallel
> ----------------------------------------------
>
> Key: HDFS-15887
> URL: https://issues.apache.org/jira/browse/HDFS-15887
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Major
> Labels: pull-request-available
> Attachments: edit_files.jpg
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and when a checkpoint occurs, it will compete with TailEdits for lock (FSNamesystem#cpLock).
> Usually, it takes a long time to execute checkpoint, which will cause the size of the generated edit log file to be relatively large.
> For example, here is an actual effect:
> The StandbyCheckpointer log is triggered as follows : edit_files.jpg
> 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint because there have been 5142154 txns since the last checkpoint, which exceeds the configured threshold 1000000
> When loading an edit log with a large amount of data, the processing time will be longer. We should make the edit log size as even as possible, which is good for the operation of the system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org