You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Suresh Srinivas (JIRA)" <ji...@apache.org> on 2010/09/18 02:08:33 UTC
[jira] Created: (HDFS-1407) Use Block in DataTransferProtocol
Use Block in DataTransferProtocol
---------------------------------
Key: HDFS-1407
URL: https://issues.apache.org/jira/browse/HDFS-1407
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Fix For: 0.22.0
Attachments: HDFS-1400.trunk.patch
Currently DataTransferProtocol has methods such as:
{noformat}
public static void opReadBlock(DataOutputStream out, long blockId,
long blockGs, long blockOffset, long blockLen, String clientName,
Token<BlockTokenIdentifier> blockToken) throws IOException;
{noformat}
The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
{noformat}
public static void opReadBlock(DataOutputStream out, Block block,
long blockOffset, long blockLen, String clientName,
Token<BlockTokenIdentifier> blockToken) throws IOException;
{noformat}
With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912580#action_12912580 ]
Konstantin Shvachko commented on HDFS-1407:
-------------------------------------------
I don't think changing the order is a problem, but this will require changing versions for all protocols. So I agree lets not do that now.
The patch looks good as is.
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-1407:
----------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
I committed the change. Konstantin, thanks for the review.
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-1407:
----------------------------------
Attachment: HDFS-1400.trunk.patch
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912778#action_12912778 ]
Suresh Srinivas commented on HDFS-1407:
---------------------------------------
All the unit tests except TestHDFSTrash passed. Here is the ouput of testpatch:
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 15 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system tests framework. The patch passed system tests framework compile.
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-1407:
----------------------------------
Attachment: HDFS-1400.trunk.patch
Updated patch with latest trunk changes.
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12911110#action_12911110 ]
Suresh Srinivas commented on HDFS-1407:
---------------------------------------
> One nit: writeId() and readId() should be called in Block.write() and Block.readFields().
Current Block.write and read serialize BlockID, Block length, and Generation stamp. Calling writeId and readId will change this order to BlockID, Generation stamp and Block length. Is this fine?
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910875#action_12910875 ]
Konstantin Shvachko commented on HDFS-1407:
-------------------------------------------
Greate refactoring. One nit: writeId() and readId() should be called in Block.write() and Block.readFields().
+1 the rest looks good.
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-1407:
----------------------------------
Status: Patch Available (was: Open)
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910865#action_12910865 ]
Suresh Srinivas commented on HDFS-1407:
---------------------------------------
A block has two pieces of information:
1. Block identifier: this include Block ID and Generation Stamp.
2. Block attribute/information: length
I have introduced writeId() and readId() methods to serialize/deserialize only identifier part of Block to be used in DataTransferProtocol.
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-1407:
----------------------------------
Status: Open (was: Patch Available)
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-1407:
----------------------------------
Status: Patch Available (was: Open)
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914201#action_12914201 ]
Hudson commented on HDFS-1407:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #396 (See [https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/396/])
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch, HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1407) Use Block in DataTransferProtocol
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-1407:
----------------------------------
Hadoop Flags: (was: [Incompatible change])
> Use Block in DataTransferProtocol
> ---------------------------------
>
> Key: HDFS-1407
> URL: https://issues.apache.org/jira/browse/HDFS-1407
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: HDFS-1400.trunk.patch
>
>
> Currently DataTransferProtocol has methods such as:
> {noformat}
> public static void opReadBlock(DataOutputStream out, long blockId,
> long blockGs, long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> The client has to pass the individual elements that make block identification such as blockId and generation stamp. I propose methods with the following format:
> {noformat}
> public static void opReadBlock(DataOutputStream out, Block block,
> long blockOffset, long blockLen, String clientName,
> Token<BlockTokenIdentifier> blockToken) throws IOException;
> {noformat}
> With this, the client need not understand the internals of Block. It receives Block over RPC and sends it in DataTransferProtocol. This helps in making Block opaque to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.