You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hadoop QA (Jira)" <ji...@apache.org> on 2020/04/11 18:22:01 UTC
[jira] [Commented] (HDFS-10743) MiniDFSCluster test runtimes can be
drastically reduce
[ https://issues.apache.org/jira/browse/HDFS-10743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081499#comment-17081499 ]
Hadoop QA commented on HDFS-10743:
----------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color} | {color:red} HDFS-10743 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10743 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12823877/HDFS-10743.003.patch |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/29123/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> MiniDFSCluster test runtimes can be drastically reduce
> ------------------------------------------------------
>
> Key: HDFS-10743
> URL: https://issues.apache.org/jira/browse/HDFS-10743
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Kuhu Shukla
> Priority: Major
> Attachments: HDFS-10743.001.patch, HDFS-10743.002.patch, HDFS-10743.003.patch
>
>
> {{MiniDFSCluster}} tests have excessive runtimes. The main problem appears to be the heartbeat interval. The NN may have to wait up to 3s (default value) for all DNs to heartbeat, triggering registration, so NN can go active. Tests that repeatedly restart the NN are severely affected.
> Example for varying heartbeat intervals for {{TestFSImageWithAcl}}:
> * 3s = ~70s -- (disgusting, why I investigated)
> * 1s = ~27s
> * 500ms = ~17s -- (had to hack DNConf for millisecond precision)
> That a 4x improvement in runtime.
> 17s is still excessively long for what the test does. Further areas to explore when running tests:
> * Reduce numerous sleeps intervals in DN's {{BPServiceActor}}.
> * Ensure heartbeats and initial BR are sent immediately upon (re)registration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org