You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Eugene Koontz <ek...@hiro-tan.org> on 2012/10/10 19:44:36 UTC
Re: Review Request: GIRAPH-211: Add secure authentication to Netty IPC
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6609/
-----------------------------------------------------------
(Updated Oct. 10, 2012, 5:44 p.m.)
Review request for giraph.
Changes
-------
Same patch as described in https://issues.apache.org/jira/browse/GIRAPH-211?focusedCommentId=13473381&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13473381
Description
-------
Current limitations:
-Not tested on other than hadoop trunk
How to use:
use
-Dgiraph.useNetty=true -Dgiraph.authenticate=true
on your job's command line to use this feature.
How to compile and test:
mvn -Phadoop_trunk clean test
Works with the following test (after applying patch to trunk):
$HADOOP_RUNTIME/bin/hadoop jar $GIRAPH_DIR/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.1-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true -Dgiraph.authenticate=true -e 2 -s 10 -v -V 2 -w 2
This addresses bug GIRAPH-211.
https://issues.apache.org/jira/browse/GIRAPH-211
Diffs (updated)
-----
pom.xml 51944fa
src/main/java/org/apache/giraph/GiraphConfiguration.java 797ba6e
src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1849504
src/main/java/org/apache/giraph/comm/CommunicationsInterface.java d60f512
src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java af9720e
src/main/java/org/apache/giraph/comm/WorkerClient.java c3ec4fe
src/main/java/org/apache/giraph/comm/netty/NettyClient.java 93f5671
src/main/java/org/apache/giraph/comm/netty/NettyServer.java 886d68b
src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 5b71401
src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientServer.java 4710f3a
src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java PRE-CREATION
src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java PRE-CREATION
src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java 3174447
src/main/java/org/apache/giraph/comm/netty/handler/SaslClientHandler.java PRE-CREATION
src/main/java/org/apache/giraph/comm/netty/handler/SaslServerHandler.java PRE-CREATION
src/main/java/org/apache/giraph/comm/requests/RequestType.java b412aa2
src/main/java/org/apache/giraph/comm/requests/SaslCompleteRequest.java PRE-CREATION
src/main/java/org/apache/giraph/comm/requests/SaslTokenMessageRequest.java PRE-CREATION
src/main/java/org/apache/giraph/graph/BspServiceWorker.java bb498ce
src/test/java/org/apache/giraph/comm/SaslConnectionTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/6609/diff/
Testing
-------
Tested only on Hadoop Trunk : needs testing on other Hadoops.
Thanks,
Eugene Koontz
Re: Review Request: GIRAPH-211: Add secure authentication to Netty IPC
Posted by Eugene Koontz <ek...@hiro-tan.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6609/
-----------------------------------------------------------
(Updated Oct. 10, 2012, 5:49 p.m.)
Review request for giraph.
Changes
-------
updated description
Description (updated)
-------
Passes
mvn -Phadoop_non_secure clean verify && mvn -Phadoop_0.20.203 clean verify && mvn -Phadoop_1.0 clean verify && mvn -Phadoop0.23 clean verify && mvn -Phadoop_2.0.0 clean verify && mvn -Phadoop_2.0.1 clean verify && mvn -Phadoop_2.0.2 clean verify
How to use:
use
-Dgiraph.useNetty=true -Dgiraph.authenticate=true
on your job's command line to use this feature.
How to compile and test (using Hadoop 2.0.0 for example):
mvn -Phadoop_2.0.0 clean test
$HADOOP_RUNTIME/bin/hadoop jar $GIRAPH_DIR/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true -Dgiraph.authenticate=true -e 100 -s 100 -v -V 100 -w 6
Look for something similar to the following in the workers' syslogs:
2012-10-10 09:24:49,924 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.netty.handler.SaslClientHandler: decode: Got a response of type SASL_TOKEN_MESSAGE_REQUEST from server:Eugenes-MacBook-Pro.local/172.16.175.1:30010
2012-10-10 09:24:49,924 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.netty.handler.SaslClientHandler: handleUpstream: Responding to server's token of length: 40
2012-10-10 09:24:49,924 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.netty.handler.SaslClientHandler: handleUpstream: Response to server is null: authentication should now be complete.
2012-10-10 09:24:49,925 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.netty.ByteCounter: handleUpstream: 0 buffer size = 17, total bytes = 207
2012-10-10 09:24:49,925 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.netty.handler.SaslClientHandler: decode: Got a response of type SASL_COMPLETE_REQUEST from server:Eugenes-MacBook-Pro.local/172.16.175.1:30010
2012-10-10 09:24:49,925 DEBUG [New I/O client worker #1-2] org.apache.giraph.comm.netty.handler.SaslClientHandler: handleUpstream: Server has sent us the SaslComplete message. Allowing normal work to proceed.
2012-10-10 09:24:49,926 DEBUG [main] org.apache.giraph.comm.netty.NettyClient: authenticateOnChannel: Authentication on channel: [id: 0x32486cdd, /172.16.175.1:54768 => Eugenes-MacBook-Pro.local/172.16.175.1:30010] has completed successfully.
..
..
2012-10-10 09:24:56,844 DEBUG [New I/O server worker #1-6] org.apache.giraph.comm.netty.handler.AuthorizeServerHandler: messageReceived: Got class org.apache.giraph.comm.requests.SendWorkerMessagesRequest
2012-10-10 09:24:56,844 DEBUG [New I/O server worker #1-6] org.apache.giraph.comm.netty.handler.AuthorizeServerHandler: messageReceived: authenticated client: FmpvYl8xMzQ5NzE1MTIxNDc1XzAyMDU= is authorized to do request on server.
2012-10-10 09:24:56,844 INFO [main] org.apache.giraph.comm.netty.NettyClient: waitAllRequests: Finished all requests. MBytes/sec sent = 0.3824, MBytes/sec received = 0.0124, MBytesSent = 0.0019, MBytesReceived = 0.0001, ave sent req MBytes = 0.0004, ave received req MBytes = 0, secs waited = 0.0040
2012-10-10 09:24:56,844 DEBUG [New I/O server worker #1-6] org.apache.giraph.comm.netty.handler.RequestServerHandler: messageReceived: Processing client 3, requestId 101, SEND_WORKER_MESSAGES_REQUEST took 30000 ns
This addresses bug GIRAPH-211.
https://issues.apache.org/jira/browse/GIRAPH-211
Diffs
-----
pom.xml 51944fa
src/main/java/org/apache/giraph/GiraphConfiguration.java 797ba6e
src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1849504
src/main/java/org/apache/giraph/comm/CommunicationsInterface.java d60f512
src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java af9720e
src/main/java/org/apache/giraph/comm/WorkerClient.java c3ec4fe
src/main/java/org/apache/giraph/comm/netty/NettyClient.java 93f5671
src/main/java/org/apache/giraph/comm/netty/NettyServer.java 886d68b
src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 5b71401
src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientServer.java 4710f3a
src/main/java/org/apache/giraph/comm/netty/SaslNettyClient.java PRE-CREATION
src/main/java/org/apache/giraph/comm/netty/SaslNettyServer.java PRE-CREATION
src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java 3174447
src/main/java/org/apache/giraph/comm/netty/handler/SaslClientHandler.java PRE-CREATION
src/main/java/org/apache/giraph/comm/netty/handler/SaslServerHandler.java PRE-CREATION
src/main/java/org/apache/giraph/comm/requests/RequestType.java b412aa2
src/main/java/org/apache/giraph/comm/requests/SaslCompleteRequest.java PRE-CREATION
src/main/java/org/apache/giraph/comm/requests/SaslTokenMessageRequest.java PRE-CREATION
src/main/java/org/apache/giraph/graph/BspServiceWorker.java bb498ce
src/test/java/org/apache/giraph/comm/SaslConnectionTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/6609/diff/
Testing (updated)
-------
Tested as above.
Thanks,
Eugene Koontz