You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "George Porter (JIRA)" <ji...@apache.org> on 2008/09/06 01:34:46 UTC
[jira] Created: (HADOOP-4096) Support for options within the
inter-datanode protocol
Support for options within the inter-datanode protocol
------------------------------------------------------
Key: HADOOP-4096
URL: https://issues.apache.org/jira/browse/HADOOP-4096
Project: Hadoop Core
Issue Type: New Feature
Components: dfs
Reporter: George Porter
This proposal is to extend the streaming inter-datanode protocol to support "options", in the spirit of IP or TCP options. This should make this protocol more extensible, allowing the client to include metadata along with commands. This would support efforts to include end-to-end and causal tracing into Hadoop, and hopefully other efforts as well.
Options should have a type, and be of variable length. It should be possible to include multiple options along with each datanode command. The option should apply to both the command and any data that is part of the command. If the datanode does not understand a given option, it should ignore it. Options should be sent end-to-end through intermediate datanodes, if necessary. For example, if an OP_WRITE_BLOCK command is pipelined through several machines, the options should be sent along the pipeline. Nodes along the pipeline may modify the options.
BTW, If HADOOP-4005 (concrete datanode protocol) is implemented, then it should solve this problem by simply letting the user add state to the concrete protocol class.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4096) Support for options within the
Datanode Transfer protocol
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628822#action_12628822 ]
George Porter commented on HADOOP-4096:
---------------------------------------
Yes, I meant the Datanode transfer protocol described in
DataNode.java, lines 916- 995. Sorry for the confusion.
Thanks,
George
> Support for options within the Datanode Transfer protocol
> ---------------------------------------------------------
>
> Key: HADOOP-4096
> URL: https://issues.apache.org/jira/browse/HADOOP-4096
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: George Porter
>
> This proposal is to extend the DataNode Transfer protocol (described in DataNode.java lines 916 - 995) to support "options", in the spirit of IP or TCP options. This should make this protocol more extensible, allowing the client to include metadata along with commands. This would support efforts to include end-to-end and causal tracing into Hadoop, and hopefully other efforts as well.
> Options should have a type, and be of variable length. It should be possible to include multiple options along with each datanode command. The option should apply to both the command and any data that is part of the command. If the datanode does not understand a given option, it should ignore it. Options should be sent end-to-end through intermediate datanodes, if necessary. For example, if an OP_WRITE_BLOCK command is pipelined through several machines, the options should be sent along the pipeline. Nodes along the pipeline may modify the options.
> BTW, If HADOOP-4005 (concrete datanode protocol) is implemented, then it should solve this problem by simply letting the user add state to the concrete protocol class.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4096) Support for options within the
Datanode Transfer protocol
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Porter updated HADOOP-4096:
----------------------------------
Description:
This proposal is to extend the DataNode Transfer protocol (described in DataNode.java lines 916 - 995) to support "options", in the spirit of IP or TCP options. This should make this protocol more extensible, allowing the client to include metadata along with commands. This would support efforts to include end-to-end and causal tracing into Hadoop, and hopefully other efforts as well.
Options should have a type, and be of variable length. It should be possible to include multiple options along with each datanode command. The option should apply to both the command and any data that is part of the command. If the datanode does not understand a given option, it should ignore it. Options should be sent end-to-end through intermediate datanodes, if necessary. For example, if an OP_WRITE_BLOCK command is pipelined through several machines, the options should be sent along the pipeline. Nodes along the pipeline may modify the options.
BTW, If HADOOP-4005 (concrete datanode protocol) is implemented, then it should solve this problem by simply letting the user add state to the concrete protocol class.
was:
This proposal is to extend the streaming inter-datanode protocol to support "options", in the spirit of IP or TCP options. This should make this protocol more extensible, allowing the client to include metadata along with commands. This would support efforts to include end-to-end and causal tracing into Hadoop, and hopefully other efforts as well.
Options should have a type, and be of variable length. It should be possible to include multiple options along with each datanode command. The option should apply to both the command and any data that is part of the command. If the datanode does not understand a given option, it should ignore it. Options should be sent end-to-end through intermediate datanodes, if necessary. For example, if an OP_WRITE_BLOCK command is pipelined through several machines, the options should be sent along the pipeline. Nodes along the pipeline may modify the options.
BTW, If HADOOP-4005 (concrete datanode protocol) is implemented, then it should solve this problem by simply letting the user add state to the concrete protocol class.
Summary: Support for options within the Datanode Transfer protocol (was: Support for options within the inter-datanode protocol)
Fixed terminology in the title
> Support for options within the Datanode Transfer protocol
> ---------------------------------------------------------
>
> Key: HADOOP-4096
> URL: https://issues.apache.org/jira/browse/HADOOP-4096
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: George Porter
>
> This proposal is to extend the DataNode Transfer protocol (described in DataNode.java lines 916 - 995) to support "options", in the spirit of IP or TCP options. This should make this protocol more extensible, allowing the client to include metadata along with commands. This would support efforts to include end-to-end and causal tracing into Hadoop, and hopefully other efforts as well.
> Options should have a type, and be of variable length. It should be possible to include multiple options along with each datanode command. The option should apply to both the command and any data that is part of the command. If the datanode does not understand a given option, it should ignore it. Options should be sent end-to-end through intermediate datanodes, if necessary. For example, if an OP_WRITE_BLOCK command is pipelined through several machines, the options should be sent along the pipeline. Nodes along the pipeline may modify the options.
> BTW, If HADOOP-4005 (concrete datanode protocol) is implemented, then it should solve this problem by simply letting the user add state to the concrete protocol class.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4096) Support for options within the
inter-datanode protocol
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628803#action_12628803 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4096:
------------------------------------------------
Do you mean "Datanode transfer protocol", i.e. the protocol described in DataNode.java line 916 - 995? Inter-datanode protocol is another protocol defined in org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol.
> Support for options within the inter-datanode protocol
> ------------------------------------------------------
>
> Key: HADOOP-4096
> URL: https://issues.apache.org/jira/browse/HADOOP-4096
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: George Porter
>
> This proposal is to extend the streaming inter-datanode protocol to support "options", in the spirit of IP or TCP options. This should make this protocol more extensible, allowing the client to include metadata along with commands. This would support efforts to include end-to-end and causal tracing into Hadoop, and hopefully other efforts as well.
> Options should have a type, and be of variable length. It should be possible to include multiple options along with each datanode command. The option should apply to both the command and any data that is part of the command. If the datanode does not understand a given option, it should ignore it. Options should be sent end-to-end through intermediate datanodes, if necessary. For example, if an OP_WRITE_BLOCK command is pipelined through several machines, the options should be sent along the pipeline. Nodes along the pipeline may modify the options.
> BTW, If HADOOP-4005 (concrete datanode protocol) is implemented, then it should solve this problem by simply letting the user add state to the concrete protocol class.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.