You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/03/29 09:03:05 UTC
[jira] [Created] (HDFS-1793) Add code to inspect a storage
directory with txid-based filenames
Add code to inspect a storage directory with txid-based filenames
-----------------------------------------------------------------
Key: HDFS-1793
URL: https://issues.apache.org/jira/browse/HDFS-1793
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Todd Lipcon
Assignee: Todd Lipcon
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1793) Add code to inspect a storage
directory with txid-based filenames
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HDFS-1793.
-------------------------------
Resolution: Fixed
> Add code to inspect a storage directory with txid-based filenames
> -----------------------------------------------------------------
>
> Key: HDFS-1793
> URL: https://issues.apache.org/jira/browse/HDFS-1793
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: HDFS-1793.txt, hdfs-1793.txt
>
>
> After HDFS-1538, the startup of the NN is done with the following phases:
> - Inspect storage directories
> - Formulate a "load plan", where a load plan is:
> -- A set of recovery actions (eg deleting half-uploaded checkpoint uplods)
> -- The fsimage to restore
> -- A sequence of edit logs to replay
> - Run the recovery
> - Load specified image
> - Load specified edits.
> This JIRA adds code to inspect a set of storage directories where the image and edits files are named according to the scheme designed in HDFS-1073, and formulate such a load plan.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1793) Add code to inspect a storage
directory with txid-based filenames
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1793:
------------------------------
Attachment: HDFS-1793.txt
Rebased on committed versions of 1538 and 1792.
Also changed the name FSImageCurrentStorageInspector to FSImageTransactionalStorageInspector, since "current" is always relative.
I will be committing this version of the patch to the branch, please feel free to reopen this JIRA or file a new one if you have any comments.
> Add code to inspect a storage directory with txid-based filenames
> -----------------------------------------------------------------
>
> Key: HDFS-1793
> URL: https://issues.apache.org/jira/browse/HDFS-1793
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: HDFS-1793.txt, hdfs-1793.txt
>
>
> After HDFS-1538, the startup of the NN is done with the following phases:
> - Inspect storage directories
> - Formulate a "load plan", where a load plan is:
> -- A set of recovery actions (eg deleting half-uploaded checkpoint uplods)
> -- The fsimage to restore
> -- A sequence of edit logs to replay
> - Run the recovery
> - Load specified image
> - Load specified edits.
> This JIRA adds code to inspect a set of storage directories where the image and edits files are named according to the scheme designed in HDFS-1073, and formulate such a load plan.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1793) Add code to inspect a storage
directory with txid-based filenames
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030252#comment-13030252 ]
Eli Collins commented on HDFS-1793:
-----------------------------------
Looks great. The way the load plan is abstraced is nice.
Minor comments/nits:
* Shouldn't an inprogress file in the middle of sequence of logs be an error, looks like it's not based on the test.
* FIRST_TXNID_BASED_LAYOUT_VERSION needs to be bumped
* The log in FSImage#inspectStorageDirs for a directory with no VERSION file should perhaps be a warn instead of info.
* newline need before declaration of FoundFSImage
* Wrt TODO by getLatestImage can be annotated as test-only
> Add code to inspect a storage directory with txid-based filenames
> -----------------------------------------------------------------
>
> Key: HDFS-1793
> URL: https://issues.apache.org/jira/browse/HDFS-1793
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: HDFS-1793.txt, hdfs-1793.txt
>
>
> After HDFS-1538, the startup of the NN is done with the following phases:
> - Inspect storage directories
> - Formulate a "load plan", where a load plan is:
> -- A set of recovery actions (eg deleting half-uploaded checkpoint uplods)
> -- The fsimage to restore
> -- A sequence of edit logs to replay
> - Run the recovery
> - Load specified image
> - Load specified edits.
> This JIRA adds code to inspect a set of storage directories where the image and edits files are named according to the scheme designed in HDFS-1073, and formulate such a load plan.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1793) Add code to inspect a storage
directory with txid-based filenames
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1793:
------------------------------
Hadoop Flags: [Reviewed]
> Add code to inspect a storage directory with txid-based filenames
> -----------------------------------------------------------------
>
> Key: HDFS-1793
> URL: https://issues.apache.org/jira/browse/HDFS-1793
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: HDFS-1793.txt, hdfs-1793.txt
>
>
> After HDFS-1538, the startup of the NN is done with the following phases:
> - Inspect storage directories
> - Formulate a "load plan", where a load plan is:
> -- A set of recovery actions (eg deleting half-uploaded checkpoint uplods)
> -- The fsimage to restore
> -- A sequence of edit logs to replay
> - Run the recovery
> - Load specified image
> - Load specified edits.
> This JIRA adds code to inspect a set of storage directories where the image and edits files are named according to the scheme designed in HDFS-1073, and formulate such a load plan.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1793) Add code to inspect a storage
directory with txid-based filenames
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1793:
------------------------------
Attachment: hdfs-1793.txt
This patch depends on HDFS-1538 and HDFS-1792
> Add code to inspect a storage directory with txid-based filenames
> -----------------------------------------------------------------
>
> Key: HDFS-1793
> URL: https://issues.apache.org/jira/browse/HDFS-1793
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-1793.txt
>
>
> After HDFS-1538, the startup of the NN is done with the following phases:
> - Inspect storage directories
> - Formulate a "load plan", where a load plan is:
> -- A set of recovery actions (eg deleting half-uploaded checkpoint uplods)
> -- The fsimage to restore
> -- A sequence of edit logs to replay
> - Run the recovery
> - Load specified image
> - Load specified edits.
> This JIRA adds code to inspect a set of storage directories where the image and edits files are named according to the scheme designed in HDFS-1073, and formulate such a load plan.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1793) Add code to inspect a storage
directory with txid-based filenames
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1793:
------------------------------
Description:
After HDFS-1538, the startup of the NN is done with the following phases:
- Inspect storage directories
- Formulate a "load plan", where a load plan is:
-- A set of recovery actions (eg deleting half-uploaded checkpoint uplods)
-- The fsimage to restore
-- A sequence of edit logs to replay
- Run the recovery
- Load specified image
- Load specified edits.
This JIRA adds code to inspect a set of storage directories where the image and edits files are named according to the scheme designed in HDFS-1073, and formulate such a load plan.
Affects Version/s: Edit log branch (HDFS-1073)
Fix Version/s: Edit log branch (HDFS-1073)
> Add code to inspect a storage directory with txid-based filenames
> -----------------------------------------------------------------
>
> Key: HDFS-1793
> URL: https://issues.apache.org/jira/browse/HDFS-1793
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
>
> After HDFS-1538, the startup of the NN is done with the following phases:
> - Inspect storage directories
> - Formulate a "load plan", where a load plan is:
> -- A set of recovery actions (eg deleting half-uploaded checkpoint uplods)
> -- The fsimage to restore
> -- A sequence of edit logs to replay
> - Run the recovery
> - Load specified image
> - Load specified edits.
> This JIRA adds code to inspect a set of storage directories where the image and edits files are named according to the scheme designed in HDFS-1073, and formulate such a load plan.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1793) Add code to inspect a storage
directory with txid-based filenames
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030561#comment-13030561 ]
Todd Lipcon commented on HDFS-1793:
-----------------------------------
bq. Shouldn't an inprogress file in the middle of sequence of logs be an error, looks like it's not based on the test.
Currently, it's OK after the case of multiple NN crashes, since we don't rename the inprogress logs after restarting the NN. For example:
* NN writing to edits_1_inprogress
* NN crashes after writing 10 transactions
* NN restarts, and starts writing to edits_11_inprogress
* NN rolls, renaming edits_11_inprogress to edits_11-20 (for example)
Here the edits_1_inprogress is still valid, unfortunately.
The solution would be to have the NN rename any inprogress log after it's been read, based on the number of transactions in it. Do you think this is necessary?
bq. FIRST_TXNID_BASED_LAYOUT_VERSION needs to be bumped
I'm bumping this each time I do a merge from trunk.
bq. The log in FSImage#inspectStorageDirs for a directory with no VERSION file should perhaps be a warn instead of info.
bq. newline need before declaration of FoundFSImage
Fixed.
bq. Wrt TODO by getLatestImage can be annotated as test-only
Actually this turned out to be used, so I removed this TODO, thanks.
(Since this was a post-commit review, I just committed the above changes to the branch, rather than uploading a new patch)
> Add code to inspect a storage directory with txid-based filenames
> -----------------------------------------------------------------
>
> Key: HDFS-1793
> URL: https://issues.apache.org/jira/browse/HDFS-1793
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: HDFS-1793.txt, hdfs-1793.txt
>
>
> After HDFS-1538, the startup of the NN is done with the following phases:
> - Inspect storage directories
> - Formulate a "load plan", where a load plan is:
> -- A set of recovery actions (eg deleting half-uploaded checkpoint uplods)
> -- The fsimage to restore
> -- A sequence of edit logs to replay
> - Run the recovery
> - Load specified image
> - Load specified edits.
> This JIRA adds code to inspect a set of storage directories where the image and edits files are named according to the scheme designed in HDFS-1073, and formulate such a load plan.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira