You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2010/01/11 23:14:54 UTC
[jira] Created: (HDFS-892) optionally use Avro for namenode RPC
optionally use Avro for namenode RPC
------------------------------------
Key: HDFS-892
URL: https://issues.apache.org/jira/browse/HDFS-892
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Reporter: Doug Cutting
Assignee: Doug Cutting
Fix For: 0.22.0
It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848963#action_12848963 ]
Tsz Wo (Nicholas), SZE commented on HDFS-892:
---------------------------------------------
Patch looks mostly good. Some comments:
- It looks like that all the changes are compatible changes except renaming register(..) to registerDatanode(..) in DatanodeProtocol. So the protocol version have to be updated.
- Do we need a versionID for the new NamenodeProtocols?
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848959#action_12848959 ]
Hadoop QA commented on HDFS-892:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12439604/HDFS-892.patch
against trunk revision 926469.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/280/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/280/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/280/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/280/console
This message is automatically generated.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Attachment: HDFS-892.patch
Fixed the javac warning (something got deprecated since I first wrote the patch).
I'm not sure why tests failed. The classpath seemed to be missing the hadoop-common jar?
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Status: Patch Available (was: Open)
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853713#action_12853713 ]
Doug Cutting commented on HDFS-892:
-----------------------------------
> If you require it at compile time, why bother with making it optional at run time?
What I meant by "optional" and "not at runtime" is that it's switched off by default. The strategy is, rather than trying to maintain a long-lived branch, to incrementally develop this in trunk but leave it switched off until it's been extensively tested, benchmarked, etc.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Nicholas, for reviewing this!
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Status: Open (was: Patch Available)
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853694#action_12853694 ]
Allen Wittenauer commented on HDFS-892:
---------------------------------------
> Avro was not intended to be optional at compile time, only at run time.
That seems kind of pointless to me. If you require it at compile time, why bother with making it optional at run time?
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Status: Open (was: Patch Available)
Failure was in TestReadWhileWriting, which passes locally.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Attachment: HDFS-892.patch
Here's a new, still preliminary, version that uses the @Nullable annotation from AVRO-433 and that passes a few more tests when AvroRpcEngine is enabled.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849391#action_12849391 ]
Hadoop QA commented on HDFS-892:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12439681/HDFS-892.patch
against trunk revision 926469.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/282/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/282/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/282/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/282/console
This message is automatically generated.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853593#action_12853593 ]
Sanjay Radia commented on HDFS-892:
-----------------------------------
Just noticed that this added an Avro @Nullable.
This means that Avro is not optional as stated for this Jira.
Shouldn't this have been @Nullable defined in common?
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853623#action_12853623 ]
Sanjay Radia commented on HDFS-892:
-----------------------------------
Doug, you defined:
+public interface NamenodeProtocols
+ extends ClientProtocol,
+ DatanodeProtocol,
+ NamenodeProtocol,
+ RefreshAuthorizationPolicyProtocol,
+ RefreshUserToGroupMappingsProtocol { ...}
Will this prevent us from putting the DatanodeProtocol in a different port?
Was the super-interface added to avoid the duplicate "versioned" interface?
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Attachment: HDFS-892.patch
> all the changes are compatible changes except renaming register(..) to registerDatanode(..) in DatanodeProtocol. So the protocol version have to be updated.
Thanks for catching that! I've attached a new version that updates this.
> Do we need a versionID for the new NamenodeProtocols?
I don't think so, clients are not permitted to use that interface. NameNode's getProtocolVersion() implementation dispatches depending on the client's protocol. If a client tried to connect using the NamenodeProtocols interface, then that method will fail with "unknown protocol".
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-892:
----------------------------------------
Hadoop Flags: [Incompatible change, Reviewed]
+1 patch looks good. Thanks, Doug.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Status: Open (was: Patch Available)
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853630#action_12853630 ]
Doug Cutting commented on HDFS-892:
-----------------------------------
> Q. Doug, Any reason why you didn't define the @Nullable in common?
That would only make sense if the rest of Avro's reflect code were in common, since it's the reflect code that interprets this annotation. Hadoop is the primary user of Avro reflect, but I don't think the only user.
> Will this prevent us from putting the DatanodeProtocol in a different port?
No more so than the current implementation does. I don't know what the plan is to implement multiple ports. The obvious approach would be to run multiple org.apache.hadoop.ipc.Servers, each serving different objects implementing different protocols. That would work fine with Avro too.
> Was the super-interface added to avoid the duplicate "versioned" interface?
No, the super-interface was added because Avro needs to know the full set of methods that a given service implements. But Namenode has lots of other public methods that are not protocol methods, so we can't otherwise determine the intended set of protocol methods.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Status: Patch Available (was: Open)
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Attachment: HDFS-892.patch
Here's a first version of this. Default behaviour is unchanged. Passing Ant
-Dtest.hdfs.rpc.engine=org.apache.hadoop.ipc.AvroRpcEngine will cause Avro reflection to be used on all NN protocols, and Avro-format RPC calls to be made over a Hadoop RPC tunnel. Some but not all tests yet pass in this configuration. The primary intent of this issue is to test that Avro reflect can correctly characterize all of the Namenode's protocols.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Status: Patch Available (was: Open)
Now that Avro 1.3.1 is available this should pass Hudson's tests.
This annotates & slightly modifies namenode RPC protocols so that Avro reflection can accurately induce an RPC protocol. A test is added that only tests basic operations using Avro.
Default runtime behavior of HDFS is not altered by this patch.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Attachment: HDFS-892.patch
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Status: Patch Available (was: Open)
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849509#action_12849509 ]
Hudson commented on HDFS-892:
-----------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #221 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/221/])
. Optionally use Avro reflection for Namenode RPC.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849026#action_12849026 ]
Hadoop QA commented on HDFS-892:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12439604/HDFS-892.patch
against trunk revision 926469.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/281/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/281/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/281/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/281/console
This message is automatically generated.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853621#action_12853621 ]
Sanjay Radia commented on HDFS-892:
-----------------------------------
Actually I am okay with Hadoop completely depending on Avro once we write all the protocols in Avro IDL (HDFS-1069). So I have no philosophical issues on depending on Avro.
BTW when we write the protocols in Avro IDL we will no longer need the @Nullable; correct?
Q. Doug, Any reason why you didn't define the @Nullable in common?
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848479#action_12848479 ]
Hadoop QA commented on HDFS-892:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12439528/HDFS-892.patch
against trunk revision 925509.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 25 javac compiler warnings (more than the trunk's current 24 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/132/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/132/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/132/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/132/console
This message is automatically generated.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HDFS-892:
------------------------------
Attachment: HDFS-892.patch
Here's an updated patch.
One can run tests using Avro serialization for RPC data by specifying -Dtestdfs.rpc.engine=org.apache.hadoop.ipc.AvroRpcEngine. Not all tests pass yet with this setting. I'll keep working on that, but I don't think that should block eventual commit of this patch, as we won't switch RPC to use Avro serialization when this is committed. Rather, this patch is intended to enable developers to work towards that goal without maintaining a separate branch.
Changes in this patch include:
- permit Ivy to find unreleased artifacts in the local Maven repository for non-Hadoop projects. This will be important if/when Avro becomes a TLP but could perhaps be a separate issue if folks prefer.
- renames DatanodeProtocol#register to be registerDatanode, since register is overloaded in NameNodeProtocol and NameNode implements both protocols. Avro does not permit overloading, instead permitting different method signatures for different revisions of the protocol.
- adds an @Union declaration to DatanodeCommand, listing its subclasses, so that interfaces that reference can know what types they'll need to serialize.
- adds a NamenodeProtocols interface that extends all of the protocols that the Namenode implements, so that we can refer to this as an RPC interface independently of all of the other methods in Namenode.
- declares some transient fields as such
- uses @Union to declare that some parameters and return types may sometimes be null
- changes a few things that are sometimes null but not seeingly for any good reason so that they instead have a non-null default value.
This is still a work in progress.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849523#action_12849523 ]
Hudson commented on HDFS-892:
-----------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #222 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/222/])
. Fix typo in initial CHANGES.txt message.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-892) optionally use Avro for namenode RPC
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853608#action_12853608 ]
Doug Cutting commented on HDFS-892:
-----------------------------------
Avro was not intended to be optional at compile time, only at run time.
> optionally use Avro for namenode RPC
> ------------------------------------
>
> Key: HDFS-892
> URL: https://issues.apache.org/jira/browse/HDFS-892
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Doug Cutting
> Assignee: Doug Cutting
> Fix For: 0.22.0
>
> Attachments: HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch, HDFS-892.patch
>
>
> It should be possible to configure HDFS so that Avro is used for RPCs to the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.