You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Alessandro Presta <al...@fb.com> on 2012/10/11 03:25:32 UTC
Review Request: GIRAPH-200: remove hadoop RPC and keep just netty
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7521/
-----------------------------------------------------------
Review request for giraph.
Description
-------
- Removed Hadoop RPC.
- Simplified Vertex API: since messages are not stored in Vertex anymore, now the standard way to define your own edge data structure is by implementing setEdges()/setNeighbors(). It's still possible to override initialize() if more customization is needed.
This addresses bug GIRAPH-200.
https://issues.apache.org/jira/browse/GIRAPH-200
Diffs
-----
/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexInputFormat.java 1396858
/trunk/pom.xml 1396858
/trunk/src/main/java/org/apache/giraph/GiraphConfiguration.java 1396858
/trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/RPCCommunications.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/VertexList.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/WorkerServer.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/messages/package-info.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/netty/NettyServer.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/package-info.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/LongDoubleFloatDoubleTextInputFormat.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/NormalizingLongDoubleFloatDoubleTextInputFormat.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/EdgeListVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/HashMapVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/IntIntNullIntVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleEdgeListVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/LongDoubleNullDoubleVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/MutableVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/SimpleMutableVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/SimpleVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/Vertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/VertexResolver.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/WorkerInfo.java 1396858
/trunk/src/main/java/org/apache/giraph/hadoop/BspPolicyProvider.java 1396858
/trunk/src/main/java/org/apache/giraph/hadoop/BspTokenSelector.java 1396858
/trunk/src/main/java/org/apache/giraph/hadoop/package-info.java 1396858
/trunk/src/main/java/org/apache/giraph/io/PseudoRandomVertexInputFormat.java 1396858
/trunk/src/main/java/org/apache/giraph/io/TextVertexInputFormat.java 1396858
/trunk/src/test/java/org/apache/giraph/comm/RPCCommunicationsTest.java 1396858
/trunk/src/test/java/org/apache/giraph/comm/RequestTest.java 1396858
/trunk/src/test/java/org/apache/giraph/examples/SimpleShortestPathsVertexTest.java 1396858
/trunk/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingVertexTest.java 1396858
/trunk/src/test/java/org/apache/giraph/examples/TryMultiIpcBindingPortsTest.java PRE-CREATION
/trunk/src/test/java/org/apache/giraph/examples/TryMultiRpcBindingPortsTest.java 1396858
/trunk/src/test/java/org/apache/giraph/graph/TestEdgeListVertex.java 1396858
/trunk/src/test/java/org/apache/giraph/graph/TestIntIntNullIntVertex.java 1396858
/trunk/src/test/java/org/apache/giraph/graph/partition/TestPartitionStores.java 1396858
/trunk/src/test/java/org/apache/giraph/io/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java 1396858
Diff: https://reviews.apache.org/r/7521/diff/
Testing
-------
- mvn verify (with various profiles)
- pseudo-distributed mode tests
Thanks,
Alessandro Presta
Re: Review Request: GIRAPH-200: remove hadoop RPC and keep just netty
Posted by Eugene Koontz <ek...@hiro-tan.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7521/#review12326
-----------------------------------------------------------
Looks good. Nice to see this happening!
/trunk/src/test/java/org/apache/giraph/examples/TryMultiIpcBindingPortsTest.java
<https://reviews.apache.org/r/7521/#comment26101>
Took me a minute to realize that this is a file rename - thought it was a new test at first.
- Eugene Koontz
On Oct. 11, 2012, 1:33 a.m., Alessandro Presta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7521/
> -----------------------------------------------------------
>
> (Updated Oct. 11, 2012, 1:33 a.m.)
>
>
> Review request for giraph.
>
>
> Description
> -------
>
> - Removed Hadoop RPC.
> - Simplified Vertex API: since messages are not stored in Vertex anymore, now the standard way to define your own edge data structure is by implementing setEdges()/setNeighbors(). It's still possible to override initialize() if more customization is needed.
>
>
> This addresses bug GIRAPH-200.
> https://issues.apache.org/jira/browse/GIRAPH-200
>
>
> Diffs
> -----
>
> /trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexInputFormat.java 1396858
> /trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeInputFormat.java 1396858
> /trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeInputFormat.java 1396858
> /trunk/pom.xml 1396858
> /trunk/src/main/java/org/apache/giraph/GiraphConfiguration.java 1396858
> /trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/RPCCommunications.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/VertexList.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/WorkerServer.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/messages/package-info.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/netty/NettyServer.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 1396858
> /trunk/src/main/java/org/apache/giraph/comm/package-info.java 1396858
> /trunk/src/main/java/org/apache/giraph/examples/LongDoubleFloatDoubleTextInputFormat.java 1396858
> /trunk/src/main/java/org/apache/giraph/examples/NormalizingLongDoubleFloatDoubleTextInputFormat.java 1396858
> /trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/EdgeListVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/HashMapVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/IntIntNullIntVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleEdgeListVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/LongDoubleNullDoubleVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/MutableVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/SimpleMutableVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/SimpleVertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/Vertex.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/VertexResolver.java 1396858
> /trunk/src/main/java/org/apache/giraph/graph/WorkerInfo.java 1396858
> /trunk/src/main/java/org/apache/giraph/hadoop/BspPolicyProvider.java 1396858
> /trunk/src/main/java/org/apache/giraph/hadoop/BspTokenSelector.java 1396858
> /trunk/src/main/java/org/apache/giraph/hadoop/package-info.java 1396858
> /trunk/src/main/java/org/apache/giraph/io/PseudoRandomVertexInputFormat.java 1396858
> /trunk/src/main/java/org/apache/giraph/io/TextVertexInputFormat.java 1396858
> /trunk/src/test/java/org/apache/giraph/comm/RPCCommunicationsTest.java 1396858
> /trunk/src/test/java/org/apache/giraph/comm/RequestTest.java 1396858
> /trunk/src/test/java/org/apache/giraph/examples/SimpleShortestPathsVertexTest.java 1396858
> /trunk/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingVertexTest.java 1396858
> /trunk/src/test/java/org/apache/giraph/examples/TryMultiIpcBindingPortsTest.java PRE-CREATION
> /trunk/src/test/java/org/apache/giraph/examples/TryMultiRpcBindingPortsTest.java 1396858
> /trunk/src/test/java/org/apache/giraph/graph/TestEdgeListVertex.java 1396858
> /trunk/src/test/java/org/apache/giraph/graph/TestIntIntNullIntVertex.java 1396858
> /trunk/src/test/java/org/apache/giraph/graph/partition/TestPartitionStores.java 1396858
> /trunk/src/test/java/org/apache/giraph/io/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java 1396858
>
> Diff: https://reviews.apache.org/r/7521/diff/
>
>
> Testing
> -------
>
> - mvn verify (with various profiles)
> - pseudo-distributed mode tests
>
>
> Thanks,
>
> Alessandro Presta
>
>
Re: Review Request: GIRAPH-200: remove hadoop RPC and keep just netty
Posted by Alessandro Presta <al...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7521/
-----------------------------------------------------------
(Updated Oct. 11, 2012, 1:33 a.m.)
Review request for giraph.
Changes
-------
Fixed giraph-formats-contrib.
Description
-------
- Removed Hadoop RPC.
- Simplified Vertex API: since messages are not stored in Vertex anymore, now the standard way to define your own edge data structure is by implementing setEdges()/setNeighbors(). It's still possible to override initialize() if more customization is needed.
This addresses bug GIRAPH-200.
https://issues.apache.org/jira/browse/GIRAPH-200
Diffs (updated)
-----
/trunk/giraph-formats-contrib/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexInputFormat.java 1396858
/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeInputFormat.java 1396858
/trunk/giraph-formats-contrib/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeInputFormat.java 1396858
/trunk/pom.xml 1396858
/trunk/src/main/java/org/apache/giraph/GiraphConfiguration.java 1396858
/trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/RPCCommunications.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/SecureRPCCommunications.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/VertexList.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/WorkerServer.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/messages/package-info.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/netty/NettyServer.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 1396858
/trunk/src/main/java/org/apache/giraph/comm/package-info.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/LongDoubleFloatDoubleTextInputFormat.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/NormalizingLongDoubleFloatDoubleTextInputFormat.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/EdgeListVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/HashMapVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/IntIntNullIntVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleEdgeListVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/LongDoubleNullDoubleVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/MutableVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/SimpleMutableVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/SimpleVertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/Vertex.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/VertexResolver.java 1396858
/trunk/src/main/java/org/apache/giraph/graph/WorkerInfo.java 1396858
/trunk/src/main/java/org/apache/giraph/hadoop/BspPolicyProvider.java 1396858
/trunk/src/main/java/org/apache/giraph/hadoop/BspTokenSelector.java 1396858
/trunk/src/main/java/org/apache/giraph/hadoop/package-info.java 1396858
/trunk/src/main/java/org/apache/giraph/io/PseudoRandomVertexInputFormat.java 1396858
/trunk/src/main/java/org/apache/giraph/io/TextVertexInputFormat.java 1396858
/trunk/src/test/java/org/apache/giraph/comm/RPCCommunicationsTest.java 1396858
/trunk/src/test/java/org/apache/giraph/comm/RequestTest.java 1396858
/trunk/src/test/java/org/apache/giraph/examples/SimpleShortestPathsVertexTest.java 1396858
/trunk/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingVertexTest.java 1396858
/trunk/src/test/java/org/apache/giraph/examples/TryMultiIpcBindingPortsTest.java PRE-CREATION
/trunk/src/test/java/org/apache/giraph/examples/TryMultiRpcBindingPortsTest.java 1396858
/trunk/src/test/java/org/apache/giraph/graph/TestEdgeListVertex.java 1396858
/trunk/src/test/java/org/apache/giraph/graph/TestIntIntNullIntVertex.java 1396858
/trunk/src/test/java/org/apache/giraph/graph/partition/TestPartitionStores.java 1396858
/trunk/src/test/java/org/apache/giraph/io/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java 1396858
Diff: https://reviews.apache.org/r/7521/diff/
Testing
-------
- mvn verify (with various profiles)
- pseudo-distributed mode tests
Thanks,
Alessandro Presta