You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org> on 2008/04/25 01:45:57 UTC
[jira] Created: (HADOOP-3310) Lease recovery for append
Lease recovery for append
-------------------------
Key: HADOOP-3310
URL: https://issues.apache.org/jira/browse/HADOOP-3310
Project: Hadoop Core
Issue Type: New Feature
Components: dfs
Reporter: Tsz Wo (Nicholas), SZE
In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080528b.patch
3310_20080528b.patch: fixed a bug and it passed all tests in my machine.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080522b.patch
3310_20080522b.patch:
- fixed the typo, item 2 and item 3 mentioned above
- added targets in INodeFileUnderConstruction
- renew lease even if all targets are not avaliable
- In FSDataset.updateBlock, unpdate oldblock.generationStamp before using it.
- In LeaseManager.syncBlock, if successList.isEmpty(), don't commit the block.
- In FSNamesystem.commitBlockSynchronization, don't write to editLog since the finalizeINodeFileUnderConstruction(...) has already done it.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601220#action_12601220 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3310:
------------------------------------------------
The failed test (TestIndexedSort) is not related in this issue, see HADOOP-3442
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080528c.patch
3310_20080528c.patch: fixed a problem when the last block is empty.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599503#action_12599503 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3310:
------------------------------------------------
I tried to test the patch for lease expiry. It does not work yet since we still write block to a tmp file first. FSDataset.validateBlockFile() will fail during lease recovery.
Also, FSDataset.volumeMap should use block id as key, instead of block (which compares both id and generation stamp) since the generation stamp may be not known.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601927#action_12601927 ]
Hudson commented on HADOOP-3310:
--------------------------------
Integrated in Hadoop-trunk #511 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/511/])
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080520.patch
3310_20080520.patch: Thanks, Dhruba.
- added FSDataset.interruptOngoingCreates which is invoked in FSDataset.updateBlock.
- The patch passed all unit tests
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598932#action_12598932 ]
dhruba borthakur commented on HADOOP-3310:
------------------------------------------
I see a compilation error while using the latest patch.
javac] /export/home/dhruba/snow/src/test/org/apache/hadoop/dfs/TestFileCreation.java:42: cannot find symbol
[javac] symbol : class TestFileCreation2
[javac] location: class org.apache.hadoop.dfs.TestFileCreation
[javac] static final String DIR = "/" + TestFileCreation2.class.getSimpleName() + "/";
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598472#action_12598472 ]
dhruba borthakur commented on HADOOP-3310:
------------------------------------------
One comment: The primary datanode makes an RPC call to the secondary datanode(s) to stamp the generationStamp for a block. As part of processing this request, the secondary datanode(s) should first terminate any threads that are currently writing to that block before returning "success" to this RPC. The threads that are currently writing to a block can be found in FSDataset.ActiveFile.threads.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080519.patch
3310_20080519.patch: a completed version for reviewing. Still need more tests.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080516c.patch
3310_20080516c.patch: cleanup some codes
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601685#action_12601685 ]
dhruba borthakur commented on HADOOP-3310:
------------------------------------------
Ok, 100 nodes sound good. I will commit it.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Status: Open (was: Patch Available)
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080528.patch
3310_20080528.patch: passes all tests in TestFileCreation (with a get-around of HADOOP-3453)
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3310:
-------------------------------------
Hadoop Flags: [Incompatible change, Reviewed] (was: [Incompatible change])
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596649#action_12596649 ]
dhruba borthakur commented on HADOOP-3310:
------------------------------------------
Now that HADOOP-2656 has been committed, this issue is the next in line that is required for appends. Nicholas: if you are working on this this, feel free to upload a very early version of the patch so that we can review it earlier. Thanks.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080522c.patch
3310_20080522c.patch: updated javadoc
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599099#action_12599099 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3310:
------------------------------------------------
> I see a compilation error while using the latest patch.
yeah, there is a typo: TestFileCreation2 => TestFileCreation
>... these two RPCs should also be available thru the ClientProtocol. ...
But ClientProtocol is for client-namenode communication. I think we need a new RPC recoverBlock(...) in either ClientProtocol or a new client-datanode protocol.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Release Note:
Implemented Lease Recovery for synchronized the last bock of a file lease. We have
- Added ClientDatanodeProtocol for client trigging block recovery
- Changed DatanodeProtocol to support block synchronization
- Changed InterDatanodeProtocol to support block update
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080529c.patch
3310_20080529c.patch: fixed findbugs warning.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600223#action_12600223 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3310:
------------------------------------------------
Hi Dhruba, I will fix TestFileCreation and figure out how to update a tmp file. Thank you for your comments.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080521.patch
3310_20080521.patch: improved javadoc
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Status: Open (was: Patch Available)
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Hadoop Flags: [Incompatible change]
Status: Patch Available (was: Open)
Passed all tests locally. Try hudson.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3310:
-------------------------------------
Attachment: 3310_20080524_dhruba.patch
Hi Nicholas, I took your latest patch and made changes to it so that the same lease recovery code is called from the client. It passes all unit tests except TestFileCreation. Maybe we can use this patch for further development and debugging. Also, pl feel free to make any changes to the code I added.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597352#action_12597352 ]
dhruba borthakur commented on HADOOP-3310:
------------------------------------------
The processing of DNA_RECOVERBLOCK would entail making RPCs to other datanode(s), right? This should be done in a thread that is separate from the offerService thread.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080516b.patch
3310_20080516b.patch: my latest codes
*Question*: When updating a block (i.e. updating generation stamp and block length), what happens if a reader tries to read the block? I guess the reader should get an exception. However, how to tells whether a block is being updated?
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080523.patch
3310_20080523.patch: latest codes but it fails on TestFileCreation.testFileCreationNamenodeRestart()
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3310) Lease recovery for
append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601220#action_12601220 ]
szetszwo edited comment on HADOOP-3310 at 5/30/08 2:32 PM:
-------------------------------------------------------------------------
The failed TestIndexedSort is not related to this issue, see HADOOP-3471.
was (Author: szetszwo):
The failed test (TestIndexedSort) is not related to this issue, see HADOOP-3442.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Status: Patch Available (was: Open)
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Status: Patch Available (was: Open)
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080519b.patch
3310_20080519b.patch: added a test and a few append methods in ClientProtocol, NameNode, FSNamesystem for testing. Still need more tests.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3310) Lease recovery for
append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601220#action_12601220 ]
szetszwo edited comment on HADOOP-3310 at 5/30/08 12:02 PM:
--------------------------------------------------------------------------
The failed test (TestIndexedSort) is not related to this issue, see HADOOP-3442.
was (Author: szetszwo):
The failed test (TestIndexedSort) is not related in this issue, see HADOOP-3442
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597267#action_12597267 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3310:
------------------------------------------------
Should we start a new thread to recover block in DataNode (i.e. the case DatanodeProtocol.DNA_RECOVERBLOCK)?
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3310:
-------------------------------------
Attachment: 3310_20080529.patch
This patch invokes the lease recovery code from the dfs client. It passes all unit tests.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599659#action_12599659 ]
dhruba borthakur commented on HADOOP-3310:
------------------------------------------
Hi Nicholas, the more I think of this, the more it sounds logical to make FSDataset.updateBlock work correctly if the block is either in the volumeMap or in the ongoingCreates.
Even when "append" is supported, It makes sense to keep the blocks that are currently being written to in the tmpdir. This ensures that a block report will not report these blocks. It also ensures that the periodic block scanner will not operate on these blocks. It is also an indirect persistence representation of blocks that need recovery if the datanode restarts. Can this be done?
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Assignee: Tsz Wo (Nicholas), SZE
The lease recovery algorithm is:
* 1) Namenode retrieves lease information
* 2) For each file f in the lease, consider the last block b of f
* 2.1) Get the datanodes which containing b
* 2.2) Assign one of the datanodes as the primary datanode p
* 2.3) p checks with all the datanodes,
* see whether block synchronization is required.
* 2.4) If synchronization is required
* 2.4.1) obtain a new generation stamp form the namenode
* 2.4.2) compute the minimum block length (this is the synchronized length)
* 2.4.3) update all datanodess (each datanode update its block file)
* 2.4.4) acknowledge the namenode.
* 2.5) Namenode commit the new generation stamp
* 2.6) Namenode removes f (and update INode) from the lease
* 3) Namenode removes the lease once all files have been removed.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601537#action_12601537 ]
dhruba borthakur commented on HADOOP-3310:
------------------------------------------
If this patch passes random-writer/sort on a reasonable size cluster (e.g. 500 nodes), it will be ready for "commit".
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599101#action_12599101 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3310:
------------------------------------------------
> 2. internalReleaseLease invokes lease.renew(). Instead, LeaseManager.removeExpiredLease() should invoke lease.renew(). The reason being that a lease actually corresponds to multiple files.
>
> 3. removeExpiredLease is also invoked from startFileInternal. In this case, only one file in the lease should be recovered. The current code recovers all the files in the lease.
Then, removeExpiredLease is not useful anymore since the uses of it are different in startFileInternal and LeaseManager.Monitor.run(). I will remove it and fix the caller's codes individually.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601218#action_12601218 ]
Hadoop QA commented on HADOOP-3310:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383056/3310_20080529c.patch
against trunk revision 661771.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2523/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2523/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2523/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2523/console
This message is automatically generated.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598938#action_12598938 ]
dhruba borthakur commented on HADOOP-3310:
------------------------------------------
Other issues that came to my mind:
1. I am making changes to the DFDClient. When the DFSClient encounters an error in the pipeline, it eliminates the bad node from the pipeline and needs to stamp all known good replicas with the new generation stamp. The DFSClient will invoke LeaseManager.recoverBlock. This method make a two RPC calls to the namenode : getNextGenerationStamp and commitBlockSynchronization. These two methods are part of the DataodeProtocol. The problem is that when this is invoked by the DFSClient, these two RPCs should also be available thru the ClientProtocol. Can this be arranged?
2. internalReleaseLease invokes lease.renew(). Instead, LeaseManager.removeExpiredLease() should invoke lease.renew(). The reason being that a lease actually corresponds to multiple files.
3. removeExpiredLease is also invoked from startFileInternal. In this case, only one file in the lease should be recovered. The current code recovers all the files in the lease.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598011#action_12598011 ]
dhruba borthakur commented on HADOOP-3310:
------------------------------------------
every read request has the blockid and the generation stamp. if the datanode cannot find the block (because the generation stamp has changed), then it will return an exception.
The current code also behaves as folows: When a client gets an exception, it retries other replicas. If all these replicas fail, then it goes back to the namenode to re-retrieve block locations. Now, it should get the correct generation stamp of the block. Then, the client will retry the read request to the datanode and this one shud succeed.
do you think that this will work?
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3310:
------------------------------------
Release Note: Implemented Lease Recovery to sync the last bock of a file. Added ClientDatanodeProtocol for client trigging block recovery. Changed DatanodeProtocol to support block synchronization. Changed InterDatanodeProtocol to support block update. (was: Implemented Lease Recovery for synchronized the last bock of a file lease. We have
- Added ClientDatanodeProtocol for client trigging block recovery
- Changed DatanodeProtocol to support block synchronization
- Changed InterDatanodeProtocol to support block update)
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598020#action_12598020 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3310:
------------------------------------------------
>how to tells whether a block is being updated?
When updating a block, the meta file is renamed to a tmp file at first. After the update is done, the tmp file will be renamed to the new meta file (with the new generation stamp.) It should work.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080527.patch
3310_20080527.patch:
- FSDataset.updateBlock finds block file in from both volumeMap and ongoingCreates
- TestFileCreation2 is a temporary test for running testLeaseExpireHardLimit() alone.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3310:
-------------------------------------
Attachment: 3310_20080529b.patch
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3310:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.18.0
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Status: Resolved (was: Patch Available)
I just committed it. Thanks Nicholas!
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3310:
-------------------------------------------
Attachment: 3310_20080514.patch
3310_20080514.patch: Implementing the Lease Recovery Algorithm
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600296#action_12600296 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3310:
------------------------------------------------
The failing TestFileCreation may be caused by HADOOP-3453.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600937#action_12600937 ]
Hadoop QA commented on HADOOP-3310:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383040/3310_20080529b.patch
against trunk revision 661462.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2521/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2521/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2521/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2521/console
This message is automatically generated.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3310) Lease recovery for
append
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594365#action_12594365 ]
szetszwo edited comment on HADOOP-3310 at 5/13/08 4:10 PM:
-------------------------------------------------------------------------
*Lease Recovery Algorithm*
{code}
/*
* 1) Namenode retrieves lease information
* 2) For each file f in the lease, consider the last block b of f
* 2.1) Get the datanodes which contains b
* 2.2) Assign one of the datanodes as the primary datanode p
* 2.3) p obtains a new generation stamp form the namenode
* 2.4) p get the block info from each datanode
* 2.5) p computes the minimum block length
* 2.6) p updates the datanodes, which have a valid generation stamp,
* with the new generation stamp and the minimum block length
* 2.7) p acknowledges the namenode the update results
* 2.8) Namenode updates the BlockInfo
* 2.9) Namenode removes f from the lease
* and removes the lease once all files have been removed
* 2.10) Namenode commit changes to edit log
*/
{code}
was (Author: szetszwo):
The lease recovery algorithm is:
* 1) Namenode retrieves lease information
* 2) For each file f in the lease, consider the last block b of f
* 2.1) Get the datanodes which containing b
* 2.2) Assign one of the datanodes as the primary datanode p
* 2.3) p checks with all the datanodes,
* see whether block synchronization is required.
* 2.4) If synchronization is required
* 2.4.1) obtain a new generation stamp form the namenode
* 2.4.2) compute the minimum block length (this is the synchronized length)
* 2.4.3) update all datanodess (each datanode update its block file)
* 2.4.4) acknowledge the namenode.
* 2.5) Namenode commit the new generation stamp
* 2.6) Namenode removes f (and update INode) from the lease
* 3) Namenode removes the lease once all files have been removed.
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3310) Lease recovery for append
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601679#action_12601679 ]
Mukund Madhugiri commented on HADOOP-3310:
------------------------------------------
I did not have resources to do a 500 node run. Here are the results for a 100 node run. Please let me know that works?
Sort on 100 nodes with *trunk*: time in minutes
- Random Writer: 15.96
- Sort: 52.92
- Validation: 11.94
Sort on 100 nodes with *trunk + patch*: time in minutes
- Random Writer: 14.76
- Sort: 53.76
- Validation: 11.46
> Lease recovery for append
> -------------------------
>
> Key: HADOOP-3310
> URL: https://issues.apache.org/jira/browse/HADOOP-3310
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3310_20080514.patch, 3310_20080516b.patch, 3310_20080516c.patch, 3310_20080519.patch, 3310_20080519b.patch, 3310_20080520.patch, 3310_20080521.patch, 3310_20080522b.patch, 3310_20080522c.patch, 3310_20080523.patch, 3310_20080524_dhruba.patch, 3310_20080527.patch, 3310_20080528.patch, 3310_20080528b.patch, 3310_20080528c.patch, 3310_20080529.patch, 3310_20080529b.patch, 3310_20080529c.patch
>
>
> In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.
> For more details, see the documentation in HADOOP-1700.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.