You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2009/12/27 09:17:29 UTC

[jira] Created: (HDFS-855) namenode can save images in parallel to all directories in fs.name.dir

namenode can save images in parallel to all directories in fs.name.dir
----------------------------------------------------------------------

                 Key: HDFS-855
                 URL: https://issues.apache.org/jira/browse/HDFS-855
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: name-node
            Reporter: dhruba borthakur


The namenode restart times can be reduced if the namenode can save its image to multiple directories (specified in fs.name.dir) in parallel.

The NN has a 6 GB fsimage and 1 MB edits file. The NN needed 10 minutes to load fsimage/edits into memory. It needs 7 minutes to read the 6 GB image. There are two directories in fs.name.dir.  it takes about 2 minute to save the image into first directory and another 2 minutes to save the image to the second directory. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HDFS-855) namenode can save images in parallel to all directories in fs.name.dir

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795375#action_12795375 ] 

Eli Collins commented on HDFS-855:
----------------------------------

Is loading/saving the fsimage and edits log IO bound? If so than we should consider compressing before storing.

> namenode can save images in parallel to all directories in fs.name.dir
> ----------------------------------------------------------------------
>
>                 Key: HDFS-855
>                 URL: https://issues.apache.org/jira/browse/HDFS-855
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>
> The namenode restart times can be reduced if the namenode can save its image to multiple directories (specified in fs.name.dir) in parallel.
> The NN has a 6 GB fsimage and 1 MB edits file. The NN needed 10 minutes to load fsimage/edits into memory. It needs 7 minutes to read the 6 GB image. There are two directories in fs.name.dir.  it takes about 2 minute to save the image into first directory and another 2 minutes to save the image to the second directory. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HDFS-855) namenode can save images in parallel to all directories in fs.name.dir

Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795534#action_12795534 ] 

Allen Wittenauer commented on HDFS-855:
---------------------------------------

Ops folks will need tools to edit the fsimage/edits when things go wrong if they are stored as compressed content.

> namenode can save images in parallel to all directories in fs.name.dir
> ----------------------------------------------------------------------
>
>                 Key: HDFS-855
>                 URL: https://issues.apache.org/jira/browse/HDFS-855
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>
> The namenode restart times can be reduced if the namenode can save its image to multiple directories (specified in fs.name.dir) in parallel.
> The NN has a 6 GB fsimage and 1 MB edits file. The NN needed 10 minutes to load fsimage/edits into memory. It needs 7 minutes to read the 6 GB image. There are two directories in fs.name.dir.  it takes about 2 minute to save the image into first directory and another 2 minutes to save the image to the second directory. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (HDFS-855) namenode can save images in parallel to all directories in fs.name.dir

Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan resolved HDFS-855.
------------------------------

    Resolution: Duplicate

> namenode can save images in parallel to all directories in fs.name.dir
> ----------------------------------------------------------------------
>
>                 Key: HDFS-855
>                 URL: https://issues.apache.org/jira/browse/HDFS-855
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>
> The namenode restart times can be reduced if the namenode can save its image to multiple directories (specified in fs.name.dir) in parallel.
> The NN has a 6 GB fsimage and 1 MB edits file. The NN needed 10 minutes to load fsimage/edits into memory. It needs 7 minutes to read the 6 GB image. There are two directories in fs.name.dir.  it takes about 2 minute to save the image into first directory and another 2 minutes to save the image to the second directory. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.