[jira] [Commented] (HDFS-5498) Improve datanode startup time

["Kihwal Lee (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825886#comment-13825886 ] 

Kihwal Lee commented on HDFS-5498:
----------------------------------

For quick restart (e.g. rolling upgrade), the initialization of {{BlockPoolSlice}} can be made fater by cashing {{dfsUsed}}.  Currently DU unconditionally runs the du command in its constructor. Filed HADOOP-10111 to add an option to specify an initial "used" value and delay the initial scan.

When I incorporated this change in {{BlockPoolSlice}}, the initialization time for each volume became several milliseconds on restart. It looks for the cache file and use the value if it is less than 5 minutes old (the default DU interval).

Another part of start-up time improvement will come from parallelizing  the scanning & popuating {{ReplicaMap}}.

> Improve datanode startup time
> -----------------------------
>
>                 Key: HDFS-5498
>                 URL: https://issues.apache.org/jira/browse/HDFS-5498
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kihwal Lee
>
> Similarly to HDFS-5027, an improvement  can be made for getVomeMap(). This is the phase in which ReplicaMap.is populated.  But it will be even better if datanode scans only once and do both.



--
This message was sent by Atlassian JIRA
(v6.1#6144)