You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/05/31 02:30:47 UTC
[jira] [Created] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
1073: remove/archive unneeded/old storage files
-----------------------------------------------
Key: HDFS-2016
URL: https://issues.apache.org/jira/browse/HDFS-2016
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Todd Lipcon
Assignee: Todd Lipcon
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HDFS-2016.
-------------------------------
Resolution: Fixed
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-2016.txt, hdfs-2016.txt, hdfs-2016.txt
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Rajiv Chittajallu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042640#comment-13042640 ]
Rajiv Chittajallu commented on HDFS-2016:
-----------------------------------------
By storage directories, do you mean dfs.data.dir list? If so is this jira for creating something like "lost+found" in hdfs?
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-2016.txt
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2016:
------------------------------
Component/s: name-node
Description: This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
Affects Version/s: Edit log branch (HDFS-1073)
Fix Version/s: Edit log branch (HDFS-1073)
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043045#comment-13043045 ]
Todd Lipcon commented on HDFS-2016:
-----------------------------------
Hi Rajiv. No, this is about the image and edits files inside of dfs.name.dir/current/. On the HDFS-1073 branch (of which this ticket is a part), existing files are not modified or deleted as checkpoints and log rolls happen. So, without this patch, the current/ directory will slowly fill up with old files until you're out of disk space :)
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-2016.txt
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041927#comment-13041927 ]
Eli Collins commented on HDFS-2016:
-----------------------------------
Minor comments follow, otherwise looks great.
* IMO it would be more clear to call archiveOldStorage explicitly from the NN (saveFSImage..) and 2NN (saveDigestAndRename..) paths instead of having this run as a side-effect of renameCheckpoint (which doesn't seem directly related).
* inspectStorageDirs could use a javadoc
* The javadoc for getImageTxIdToRetain needs to be finished
* Would rename retain.checkpoints.count something like num.checkpoints.retained
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-2016.txt
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2016:
------------------------------
Attachment: hdfs-2016.txt
Rebased on current branch. I'm going to commit this version. Feel free to post-commit review if there are further comments.
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-2016.txt, hdfs-2016.txt, hdfs-2016.txt
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068120#comment-13068120 ]
Eli Collins commented on HDFS-2016:
-----------------------------------
+1 to the latest patch. Thanks for addressing my comments.
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-2016.txt, hdfs-2016.txt, hdfs-2016.txt
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2016:
------------------------------
Attachment: hdfs-2016.txt
Here's a basic patch.
There are a couple issues still to work out with how this interacts with the 2NN, but the basic structure should be good.
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-2016.txt
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2016:
------------------------------
Hadoop Flags: [Reviewed]
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-2016.txt, hdfs-2016.txt, hdfs-2016.txt
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2016) 1073: remove/archive unneeded/old
storage files
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2016:
------------------------------
Attachment: hdfs-2016.txt
Updated patch.
I also discovered a potential bug, since the "inspectStorageDirs" had an unintended side effect of calling sd.read() in all of the configured storage directories. I don't know that it would have caused a problem, but it certainly wasn't intended. This patch splits up the storage dir inspection into two parts, so the unnecessary read()s don't happen in the archival code.
> 1073: remove/archive unneeded/old storage files
> -----------------------------------------------
>
> Key: HDFS-2016
> URL: https://issues.apache.org/jira/browse/HDFS-2016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-2016.txt, hdfs-2016.txt
>
>
> This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and "archives" them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira