You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2011/01/11 19:11:45 UTC
[jira] Created: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
----------------------------------------------------------------------------------------------------------------
Key: HDFS-1578
URL: https://issues.apache.org/jira/browse/HDFS-1578
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: 0.23.0
HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980242#action_12980242 ]
Hairong Kuang commented on HDFS-1578:
-------------------------------------
Our use case of this is to support across cluster communications. We assume that every datanode runs the same data transfer protocol in a cluster but two clusters may run a slight different data transfer protocol versions. So the client can fetch the version before any I/O starts. This is done once per client not for every DN commection.
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980250#action_12980250 ]
Todd Lipcon commented on HDFS-1578:
-----------------------------------
oh, duh. need more coffee :)
In that case, this seems pretty reasonable.
Any thoughts on the idea above about tracking it per-DN and including the version info in LocatedBlocks? Then we could support rolling upgrade of DNs between transfer protocol versions as well as the multi-cluster use case. Since this JIRA is just about first steps, it's fine to dismiss this idea as too complicated, just curious if it was considered.
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980256#action_12980256 ]
Todd Lipcon commented on HDFS-1578:
-----------------------------------
ah, we are have race conditions commenting here on this JIRA :)
bq. Todd, what's the advantage of making the RPC to NameNode?It looks intuitive to me to query a datanode to get its data transfer version
I agree, but you said above that you assume that within a cluster, the DN protocol will not vary - ie that the data transfer version is an attribute of a cluster, not of a particular node. So, as a cluster-wide attribute, it makes sense to query the master/NN.
Querying just one node and making the assumption all are the same seems a little fragile - you could easily end up in a state where some of the DNs in the cluster have version A and others have version B, and clients will be able to talk to some but not others depending on which one they happen to RPC to first. Will the DNs report their transfer version to the NN when they register to the cluster, and the NN will reject any DN registration that doesn't match?
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980223#action_12980223 ]
Todd Lipcon commented on HDFS-1578:
-----------------------------------
It seems like this could hurt efficiency for random workloads - we now have an extra RPC round trip (with associated connection setup and teardown) for every DN connection... what's the caching plan, etc? Could we keep the version numbers associated with each DN as part of the datanode descriptor on the NN, and stuff it in the LocatedBlocks objects?
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980254#action_12980254 ]
Hairong Kuang commented on HDFS-1578:
-------------------------------------
Now I see your point. Yes, it seems to be a good idea to include the data transfer version # in DN descriptor.
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980244#action_12980244 ]
Todd Lipcon commented on HDFS-1578:
-----------------------------------
So in that case wouldn't it be an RPC to the NN, if you assume it's cluster-wide?
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980251#action_12980251 ]
Hairong Kuang commented on HDFS-1578:
-------------------------------------
Sorry I meant ClientDatanodeProtocol.
Todd, what's the advantage of making the RPC to NameNode?It looks intuitive to me to query a datanode to get its data transfer version.
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980246#action_12980246 ]
Doug Cutting commented on HDFS-1578:
------------------------------------
Todd: DataNodeProtocol is implemented by the NameNode.
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980253#action_12980253 ]
Todd Lipcon commented on HDFS-1578:
-----------------------------------
Wait -- DatanodeProtocol can't be used since it's authorized only for the datanode principals on a secure cluster. So it would either be part of ClientProtocol at the NN or ClientDatanodeProtocol at the DN.
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981647#action_12981647 ]
Todd Lipcon commented on HDFS-1578:
-----------------------------------
bq. Todd, I really like your proposal of including data transfer version # in DN's descriptor. Is there a simple way of making this work without breaking protocol compatiblity?
Hmm, you're trying to add a compatibility layer that is itself compatible with previous versions? Seems tricky, since we'd have to add a field to DatanodeInfo... so, no, no good ideas if that is a goal.
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1578) First step towards data transter
protocol compatibility: support
DatanodeProtocol#getDataTransferProtocolVersion
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980479#action_12980479 ]
Hairong Kuang commented on HDFS-1578:
-------------------------------------
I am still tempting to get the version # from a datanode simply for performance/scalability consideration. The Apache trunk now does not have the inconsistency state concern since that it requires that datanode/namenode have the same build version. I do see the risk especially fb's deployment has relaxed this restriction.
Todd, I really like your proposal of including data transfer version # in DN's descriptor. Is there a simple way of making this work without breaking protocol compatiblity?
> First step towards data transter protocol compatibility: support DatanodeProtocol#getDataTransferProtocolVersion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1578
> URL: https://issues.apache.org/jira/browse/HDFS-1578
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
>
> HADOOP-6904 allows us to handle RPC changes in a compatible way. However, we have one more protocol to take care of, the data transfer protocol, which a dfs client uses to read data from or write data to a datanode.
>
> My proposal is to add a new RPC getDataTransferVersion to DatanodeProtocol that returns the data transfer protocol version running on the datanode. A dfs client gets the datanode's version number before it reads from/writes to a datanode. With this, the dfs client could behave differently according to datanode's data transfer version. This provides a base for us to make data transfer protocol changes in a compatible way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.