You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/08/20 18:48:00 UTC
[jira] [Commented] (HDFS-15887) Make LogRoll and TailEdits execute in parallel
[ https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17582354#comment-17582354 ]
ASF GitHub Bot commented on HDFS-15887:
---------------------------------------
hadoop-yetus commented on PR #2839:
URL: https://github.com/apache/hadoop/pull/2839#issuecomment-1221385156
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 18s | | https://github.com/apache/hadoop/pull/2839 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/2839 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2839/1/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Make LogRoll and TailEdits execute in parallel
> ----------------------------------------------
>
> Key: HDFS-15887
> URL: https://issues.apache.org/jira/browse/HDFS-15887
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Major
> Labels: pull-request-available
> Attachments: edit_files.jpg
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and when a checkpoint occurs, it will compete with TailEdits for lock (FSNamesystem#cpLock).
> Usually, it takes a long time to execute checkpoint, which will cause the size of the generated edit log file to be relatively large.
> For example, here is an actual effect:
> The StandbyCheckpointer log is triggered as follows : edit_files.jpg
> 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint because there have been 5142154 txns since the last checkpoint, which exceeds the configured threshold 1000000
> When loading an edit log with a large amount of data, the processing time will be longer. We should make the edit log size as even as possible, which is good for the operation of the system.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org