You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org> on 2008/04/19 01:24:21 UTC
[jira] Created: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Need a mechanism for data nodes to update generation stamps.
------------------------------------------------------------
Key: HADOOP-3283
URL: https://issues.apache.org/jira/browse/HADOOP-3283
Project: Hadoop Core
Issue Type: New Feature
Reporter: Tsz Wo (Nicholas), SZE
For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3283:
-------------------------------------------
Release Note:
Incompatible changes:
- Added an IPC server in DataNode and a new IPC protocol InterDatanodeProtocol. Added conf properties dfs.datanode.ipc.address and dfs.datanode.handler.count with defaults "0.0.0.0:50020" and 3, respectively.
- Changed the serialization in DatanodeRegistration and DatanodeInfo, and therefore, updated the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol.
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch, 3283_20080425versionID.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3283:
-------------------------------------------
Status: Patch Available (was: Open)
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591041#action_12591041 ]
dhruba borthakur commented on HADOOP-3283:
------------------------------------------
+1.
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594884#action_12594884 ]
Hudson commented on HADOOP-3283:
--------------------------------
Integrated in Hadoop-trunk #483 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/483/])
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch, 3283_20080425versionID.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3283:
------------------------------------
Release Note:
Added an IPC server in DataNode and a new IPC protocol InterDatanodeProtocol. Added conf properties dfs.datanode.ipc.address and dfs.datanode.handler.count with defaults "0.0.0.0:50020" and 3, respectively.
Changed the serialization in DatanodeRegistration and DatanodeInfo, and therefore, updated the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol.
was:
Incompatible changes:
- Added an IPC server in DataNode and a new IPC protocol InterDatanodeProtocol. Added conf properties dfs.datanode.ipc.address and dfs.datanode.handler.count with defaults "0.0.0.0:50020" and 3, respectively.
- Changed the serialization in DatanodeRegistration and DatanodeInfo, and therefore, updated the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol.
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.18.0
>
> Attachments: 3283_20080423c.patch, 3283_20080424.patch, 3283_20080425versionID.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592495#action_12592495 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3283:
------------------------------------------------
The new Findbugs warning says that class doesn't override equals in the superclass. In our case, we have BlockMetaDataInfo extending Block and we don't need to define equals in BlockMetaDataInfo.
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3283:
-------------------------------------
Hadoop Flags: [Incompatible change, Reviewed]
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591926#action_12591926 ]
dhruba borthakur commented on HADOOP-3283:
------------------------------------------
1. The two new methods in the InterDatanodeProtocol should take a Block object as the first parameter.
2. The MiniDFSCluster should have a method named getDatanode.getDataNode() that returns an object of type InterDatanodeProtocol. This will allow tests to easily invoke RPCs in the InterDatanodeProtocol.
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3283:
-------------------------------------------
Attachment: 3283_20080424.patch
3283_20080424.patch: fixed the bugs and update the patch with Dhruba's comments.
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3283:
-------------------------------------------
Attachment: 3283_20080425versionID.patch
3283_20080425versionID.patch: update the versionID in ClientProtocol, DatanodeProtocol, NamenodeProtocol
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch, 3283_20080425versionID.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590655#action_12590655 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3283:
------------------------------------------------
One way is to an inter-DataNode protocol for block manipulation:
{code}
interface BlockProtocol extends VersionedProtocol {
/**
* Update the GenerationStamp of a block.
* @return true iff update was required and done successfully.
*/
public boolean updateGenerationStamp(Block block,
GenerationStamp generationstamp) throws IOException;
/**
* @return the BlockMetaDataInfo of the block.
*/
public BlockMetaDataInfo getBlockMetaDataInfo(Block block) throws IOException;
}
{code}
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592567#action_12592567 ]
Hudson commented on HADOOP-3283:
--------------------------------
Integrated in Hadoop-trunk #471 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/471/])
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch, 3283_20080425versionID.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3283:
-------------------------------------------
Attachment: 3283_20080423c.patch
3283_20080423c.patch: preview, it does not pass the tests yet.
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592243#action_12592243 ]
Hadoop QA commented on HADOOP-3283:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380882/3283_20080424.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 15 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs -1. The patch appears to introduce 1 new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2320/console
This message is automatically generated.
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3283:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Status: Resolved (was: Patch Available)
I just committed this. Thanks Nicholas!
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: 3283_20080423c.patch, 3283_20080424.patch, 3283_20080425versionID.patch
>
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591043#action_12591043 ]
dhruba borthakur commented on HADOOP-3283:
------------------------------------------
Maybe we can call it InterDatanodeProtocol to make it more explicit.
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3283) Need a mechanism for data nodes to
update generation stamps.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-3283:
-------------------------------------------
Assignee: Tsz Wo (Nicholas), SZE
For adding a Datanode RPC server, we need to store the RPC server port and distribute it. One way is to store the port in DatanodeID. I am not sure whether it is the right way to go.
> Need a mechanism for data nodes to update generation stamps.
> ------------------------------------------------------------
>
> Key: HADOOP-3283
> URL: https://issues.apache.org/jira/browse/HADOOP-3283
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
>
> For implementing file append feature (HADOOP-1700), a generation stamp is added to each block. We need a mechanism for data nodes to update generation stamps for lease recovery.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.