You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/11/29 22:27:40 UTC
[jira] [Created] (HDFS-2602) Standby needs to maintain BlockInfo
while following edits
Standby needs to maintain BlockInfo while following edits
---------------------------------------------------------
Key: HDFS-2602
URL: https://issues.apache.org/jira/browse/HDFS-2602
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Aaron T. Myers
Priority: Critical
As described in HDFS-1975:
When we close a file, or add another block to a file, we write OP_CLOSE or OP_ADD in the txn log. FSEditLogLoader, when it sees these types of transactions, creates new BlockInfo objects for all of the blocks listed in the transaction. These new BlockInfos have no block locations associated. So, when we close a file, the SBNN loses its block locations info for that file and is no longer "hot".
I have an ugly hack which copies over the old BlockInfos from the existing INode, but I'm not convinced it's the right way. It might be cleaner to add new opcode types like OP_ADD_ADDITIONAL_BLOCK, and actually treat OP_CLOSE as just a finalization of INodeFileUnderConstruction to INodeFile, rather than replacing block info at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2602) NN should log newly-allocated blocks
without losing BlockInfo
Posted by "Aaron T. Myers (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers resolved HDFS-2602.
----------------------------------
Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)
Hadoop Flags: Reviewed
Thanks a lot for your many reviews, Todd. I've just committed this to the HA branch.
> NN should log newly-allocated blocks without losing BlockInfo
> -------------------------------------------------------------
>
> Key: HDFS-2602
> URL: https://issues.apache.org/jira/browse/HDFS-2602
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: HA branch (HDFS-1623)
>
> Attachments: HDFS-2602.patch, HDFS-2602.patch, HDFS-2602.patch, HDFS-2602.patch, HDFS-2602.patch
>
>
> Without the patch in HDFS-1108, new block allocations aren't logged to the edits log. For HA, we'll need that functionality and we'll need to make sure that block locations aren't blown away in the Standby NN when tailing the edits log.
> As described in HDFS-1975:
> When we close a file, or add another block to a file, we write OP_CLOSE or OP_ADD in the txn log. FSEditLogLoader, when it sees these types of transactions, creates new BlockInfo objects for all of the blocks listed in the transaction. These new BlockInfos have no block locations associated. So, when we close a file, the SBNN loses its block locations info for that file and is no longer "hot".
> I have an ugly hack which copies over the old BlockInfos from the existing INode, but I'm not convinced it's the right way. It might be cleaner to add new opcode types like OP_ADD_ADDITIONAL_BLOCK, and actually treat OP_CLOSE as just a finalization of INodeFileUnderConstruction to INodeFile, rather than replacing block info at all.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira