You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Bingjing Zhang <je...@gmail.com> on 2013/07/01 02:16:12 UTC
Re: Review Request 12174: Add "one-to-all" message sending strategy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/
-----------------------------------------------------------
(Updated July 1, 2013, 12:16 a.m.)
Review request for giraph.
Changes
-------
Refactor sendMessageCache and add sendMessageToAllCache.
Bugs: GIRAPH-701
https://issues.apache.org/jira/browse/GIRAPH-701
Repository: giraph-git
Description
-------
Add "one-to-all" message sending strategy.
Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
To enable it, use conf.enableOneToAllMsgSending()
Diffs (updated)
-----
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 40023c2
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java affc260
giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java f18af5b
giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java d786db5
giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 2d232a6
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c65b5f0
giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 9b3f165
giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java c8c09df
giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/GiraphHCatInputFormat.java 2e91cba
Diff: https://reviews.apache.org/r/12174/diff/
Testing
-------
Did
mvn clean verify
mvn clean test
Thanks,
Bingjing Zhang
Re: Review Request 12174: Add "one-to-all" message sending strategy
Posted by Bingjing Zhang <je...@gmail.com>.
> On July 8, 2013, 6:40 p.m., Avery Ching wrote:
> > giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java, lines 138-142
> > <https://reviews.apache.org/r/12174/diff/2/?file=314698#file314698line138>
> >
> > Isn't this expensive to do the copy into a format that can be processed by the same code as the original send messages? Can't we avoid a buffer copy by directly deserializing and adding to the message store directly? Also, is this compatible with changing the message store between supersteps?
Due to the synchronization issue, currently this "formatting" is the most efficient way to put messages to the message store.
> On July 8, 2013, 6:40 p.m., Avery Ching wrote:
> > giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java, lines 105-121
> > <https://reviews.apache.org/r/12174/diff/2/?file=314702#file314702line105>
> >
> > I don't think these are necessary right? Your request has access to conf.
createMessage is kepted because now message is created through messageValueFactory.createMessageValue()
- Bingjing
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/#review22838
-----------------------------------------------------------
On July 10, 2013, 9:12 p.m., Bingjing Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12174/
> -----------------------------------------------------------
>
> (Updated July 10, 2013, 9:12 p.m.)
>
>
> Review request for giraph.
>
>
> Bugs: GIRAPH-701
> https://issues.apache.org/jira/browse/GIRAPH-701
>
>
> Repository: giraph-git
>
>
> Description
> -------
>
> Add "one-to-all" message sending strategy.
> Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
> To enable it, use conf.enableOneToAllMsgSending()
>
>
> Diffs
> -----
>
> giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ff3f06d
> giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 2eeac18
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java a50f673
> giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
> giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java bedaf48
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 7ce0083
> giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 74f1ba5
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f
> giraph-core/src/main/java/org/apache/giraph/counters/GiraphStats.java 3f25508
> giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
> giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java a9bf3fd
> giraph-core/src/main/java/org/apache/giraph/graph/GlobalStats.java f3cbea2
> giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 1d3cff0
> giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac
> giraph-core/src/main/java/org/apache/giraph/partition/PartitionStats.java b8eeca9
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 26c547b
> giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 52bac3f
> giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java 2e60c09
> giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
> giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingComputationTest.java 7346745
>
> Diff: https://reviews.apache.org/r/12174/diff/
>
>
> Testing
> -------
>
> Did
> mvn clean verify
> mvn clean test
>
>
> Thanks,
>
> Bingjing Zhang
>
>
Re: Review Request 12174: Add "one-to-all" message sending strategy
Posted by Avery Ching <av...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/#review22838
-----------------------------------------------------------
This needs a bit of work, but the performance seems pretty good from your experiments. Please address the below questions/comments. Don't forget to rebase =).
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java
<https://reviews.apache.org/r/12174/#comment46546>
Since you do not change values, it is safer to make this private and then have a getter for the individual buffer sizes. I.e. getInitialBufferSize(int workerIndex) or something like that.
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java
<https://reviews.apache.org/r/12174/#comment46544>
Why did you change this. It was private with a getter before?
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java
<https://reviews.apache.org/r/12174/#comment46545>
Want to make this private with a getter as well?
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java
<https://reviews.apache.org/r/12174/#comment46554>
Can we factor out the common code here? The difference is only one line.
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java
<https://reviews.apache.org/r/12174/#comment46557>
Flush the rest of the messages to the workers
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment46550>
extra space
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment46563>
Indent.
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment46560>
So don't worry weather =>
Therefore, it doesn't matter if the result is null or not.
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment46561>
indent wrong
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment46562>
that belong
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment46551>
You must rethrow, say as an IllegalStateException. This will let errors through.
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment46564>
space.
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment46567>
This is wrong now I think.
giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
<https://reviews.apache.org/r/12174/#comment46568>
Nice job refactoring this out.
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java
<https://reviews.apache.org/r/12174/#comment46569>
extra space. Did you run with verify? It should have caught this.
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java
<https://reviews.apache.org/r/12174/#comment46552>
extra space
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java
<https://reviews.apache.org/r/12174/#comment46570>
Also, please explain the logic for the 2x buffer size.
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java
<https://reviews.apache.org/r/12174/#comment46573>
extra space!
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java
<https://reviews.apache.org/r/12174/#comment46572>
Please break line after the .
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java
<https://reviews.apache.org/r/12174/#comment46574>
Isn't this expensive to do the copy into a format that can be processed by the same code as the original send messages? Can't we avoid a buffer copy by directly deserializing and adding to the message store directly? Also, is this compatible with changing the message store between supersteps?
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java
<https://reviews.apache.org/r/12174/#comment46571>
Not necessary. You are throwing in the next line.
giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java
<https://reviews.apache.org/r/12174/#comment46575>
I don't think these are necessary right? Your request has access to conf.
giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/GiraphHCatInputFormat.java
<https://reviews.apache.org/r/12174/#comment46553>
This is not your change right? From trunk?
- Avery Ching
On July 1, 2013, 12:16 a.m., Bingjing Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12174/
> -----------------------------------------------------------
>
> (Updated July 1, 2013, 12:16 a.m.)
>
>
> Review request for giraph.
>
>
> Bugs: GIRAPH-701
> https://issues.apache.org/jira/browse/GIRAPH-701
>
>
> Repository: giraph-git
>
>
> Description
> -------
>
> Add "one-to-all" message sending strategy.
> Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
> To enable it, use conf.enableOneToAllMsgSending()
>
>
> Diffs
> -----
>
> giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 40023c2
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java affc260
> giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
> giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java f18af5b
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java d786db5
> giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 2d232a6
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c65b5f0
> giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 9b3f165
> giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java c8c09df
> giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
> giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/GiraphHCatInputFormat.java 2e91cba
>
> Diff: https://reviews.apache.org/r/12174/diff/
>
>
> Testing
> -------
>
> Did
> mvn clean verify
> mvn clean test
>
>
> Thanks,
>
> Bingjing Zhang
>
>
Re: Review Request 12174: Add "one-to-all" message sending strategy
Posted by Bingjing Zhang <je...@gmail.com>.
> On July 19, 2013, 8:27 p.m., Avery Ching wrote:
> > giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java, lines 89-149
> > <https://reviews.apache.org/r/12174/diff/4/?file=322175#file322175line89>
> >
> > Why not do this logic the same way that SendWorkerRequestMessages does (i.e. via addPartitionMessages()?
move this to next task
- Bingjing
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/#review23544
-----------------------------------------------------------
On July 16, 2013, 11:33 p.m., Bingjing Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12174/
> -----------------------------------------------------------
>
> (Updated July 16, 2013, 11:33 p.m.)
>
>
> Review request for giraph.
>
>
> Bugs: GIRAPH-701
> https://issues.apache.org/jira/browse/GIRAPH-701
>
>
> Repository: giraph-git
>
>
> Description
> -------
>
> Add "one-to-all" message sending strategy.
> Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
> To enable it, use conf.enableOneToAllMsgSending()
>
>
> Diffs
> -----
>
> giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ff3f06d
> giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 2eeac18
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java a50f673
> giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
> giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java bedaf48
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 7ce0083
> giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 21cf245
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f
> giraph-core/src/main/java/org/apache/giraph/counters/GiraphStats.java 3f25508
> giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
> giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java a9bf3fd
> giraph-core/src/main/java/org/apache/giraph/graph/GlobalStats.java f3cbea2
> giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 1d3cff0
> giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac
> giraph-core/src/main/java/org/apache/giraph/partition/PartitionStats.java b8eeca9
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 26c547b
> giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 52bac3f
> giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java 2e60c09
> giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
> giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingComputationTest.java 7346745
>
> Diff: https://reviews.apache.org/r/12174/diff/
>
>
> Testing
> -------
>
> Did
> mvn clean verify
> mvn clean test
>
>
> Thanks,
>
> Bingjing Zhang
>
>
Re: Review Request 12174: Add "one-to-all" message sending strategy
Posted by Avery Ching <av...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/#review23544
-----------------------------------------------------------
Looks pretty close. Just a couple of minor things, then let's get this in.
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java
<https://reviews.apache.org/r/12174/#comment47440>
This can be private as before with a method to get the keyset (i.e. Set<WorkerInfo> getWorkerInfos()).
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java
<https://reviews.apache.org/r/12174/#comment47441>
Note that this comment isn't necessary as per the checkstyle rules (when the method is get*, where the member variable is *).
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment47443>
missing "
i.e. "one-to-all"
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
<https://reviews.apache.org/r/12174/#comment47448>
I think I like flush being part of the cache, nice.
giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
<https://reviews.apache.org/r/12174/#comment47447>
This is fixing a bug correct? If so shouldn't we insure that it's fixed at the source? I.e. when we construct the request we should have conf in the constructor?
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java
<https://reviews.apache.org/r/12174/#comment47445>
Why not do this logic the same way that SendWorkerRequestMessages does (i.e. via addPartitionMessages()?
- Avery Ching
On July 16, 2013, 11:33 p.m., Bingjing Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12174/
> -----------------------------------------------------------
>
> (Updated July 16, 2013, 11:33 p.m.)
>
>
> Review request for giraph.
>
>
> Bugs: GIRAPH-701
> https://issues.apache.org/jira/browse/GIRAPH-701
>
>
> Repository: giraph-git
>
>
> Description
> -------
>
> Add "one-to-all" message sending strategy.
> Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
> To enable it, use conf.enableOneToAllMsgSending()
>
>
> Diffs
> -----
>
> giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ff3f06d
> giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 2eeac18
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java a50f673
> giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
> giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java bedaf48
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 7ce0083
> giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 21cf245
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f
> giraph-core/src/main/java/org/apache/giraph/counters/GiraphStats.java 3f25508
> giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
> giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java a9bf3fd
> giraph-core/src/main/java/org/apache/giraph/graph/GlobalStats.java f3cbea2
> giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 1d3cff0
> giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac
> giraph-core/src/main/java/org/apache/giraph/partition/PartitionStats.java b8eeca9
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 26c547b
> giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 52bac3f
> giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java 2e60c09
> giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
> giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingComputationTest.java 7346745
>
> Diff: https://reviews.apache.org/r/12174/diff/
>
>
> Testing
> -------
>
> Did
> mvn clean verify
> mvn clean test
>
>
> Thanks,
>
> Bingjing Zhang
>
>
Re: Review Request 12174: Add "one-to-all" message sending strategy
Posted by Avery Ching <av...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/#review23570
-----------------------------------------------------------
Ship it!
+1, we'll commit this as is, but please fix the aforementioned code cleanup before you leave this summer.
- Avery Ching
On July 20, 2013, 3:31 a.m., Bingjing Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12174/
> -----------------------------------------------------------
>
> (Updated July 20, 2013, 3:31 a.m.)
>
>
> Review request for giraph.
>
>
> Bugs: GIRAPH-701
> https://issues.apache.org/jira/browse/GIRAPH-701
>
>
> Repository: giraph-git
>
>
> Description
> -------
>
> Add "one-to-all" message sending strategy.
> Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
> To enable it, use conf.enableOneToAllMsgSending()
>
>
> Diffs
> -----
>
> giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ff3f06d
> giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 2eeac18
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java a50f673
> giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
> giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java bedaf48
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 7ce0083
> giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 21cf245
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f
> giraph-core/src/main/java/org/apache/giraph/counters/GiraphStats.java 3f25508
> giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
> giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java a9bf3fd
> giraph-core/src/main/java/org/apache/giraph/graph/GlobalStats.java f3cbea2
> giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 1d3cff0
> giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac
> giraph-core/src/main/java/org/apache/giraph/partition/PartitionStats.java b8eeca9
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 26c547b
> giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 52bac3f
> giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java 2e60c09
> giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
> giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingComputationTest.java 7346745
>
> Diff: https://reviews.apache.org/r/12174/diff/
>
>
> Testing
> -------
>
> Did
> mvn clean verify
> mvn clean test
>
>
> Thanks,
>
> Bingjing Zhang
>
>
Re: Review Request 12174: Add "one-to-all" message sending strategy
Posted by Avery Ching <av...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/#review23569
-----------------------------------------------------------
Ship it!
+1, we'll commit this as is, but please fix the aforementioned code cleanup before you leave this summer.
- Avery Ching
On July 20, 2013, 3:31 a.m., Bingjing Zhang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/12174/
> -----------------------------------------------------------
>
> (Updated July 20, 2013, 3:31 a.m.)
>
>
> Review request for giraph.
>
>
> Bugs: GIRAPH-701
> https://issues.apache.org/jira/browse/GIRAPH-701
>
>
> Repository: giraph-git
>
>
> Description
> -------
>
> Add "one-to-all" message sending strategy.
> Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
> To enable it, use conf.enableOneToAllMsgSending()
>
>
> Diffs
> -----
>
> giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ff3f06d
> giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 2eeac18
> giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java a50f673
> giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
> giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java bedaf48
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 7ce0083
> giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 21cf245
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f
> giraph-core/src/main/java/org/apache/giraph/counters/GiraphStats.java 3f25508
> giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
> giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java a9bf3fd
> giraph-core/src/main/java/org/apache/giraph/graph/GlobalStats.java f3cbea2
> giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 1d3cff0
> giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac
> giraph-core/src/main/java/org/apache/giraph/partition/PartitionStats.java b8eeca9
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 26c547b
> giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 52bac3f
> giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java 2e60c09
> giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
> giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingComputationTest.java 7346745
>
> Diff: https://reviews.apache.org/r/12174/diff/
>
>
> Testing
> -------
>
> Did
> mvn clean verify
> mvn clean test
>
>
> Thanks,
>
> Bingjing Zhang
>
>
Re: Review Request 12174: Add "one-to-all" message sending strategy
Posted by Bingjing Zhang <je...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/
-----------------------------------------------------------
(Updated July 20, 2013, 3:31 a.m.)
Review request for giraph.
Changes
-------
mvn clean verify
mvn clean install
Bugs: GIRAPH-701
https://issues.apache.org/jira/browse/GIRAPH-701
Repository: giraph-git
Description
-------
Add "one-to-all" message sending strategy.
Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
To enable it, use conf.enableOneToAllMsgSending()
Diffs (updated)
-----
giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ff3f06d
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 2eeac18
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java a50f673
giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java bedaf48
giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 7ce0083
giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 21cf245
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f
giraph-core/src/main/java/org/apache/giraph/counters/GiraphStats.java 3f25508
giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java a9bf3fd
giraph-core/src/main/java/org/apache/giraph/graph/GlobalStats.java f3cbea2
giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 1d3cff0
giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac
giraph-core/src/main/java/org/apache/giraph/partition/PartitionStats.java b8eeca9
giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 26c547b
giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 52bac3f
giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java 2e60c09
giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingComputationTest.java 7346745
Diff: https://reviews.apache.org/r/12174/diff/
Testing
-------
Did
mvn clean verify
mvn clean test
Thanks,
Bingjing Zhang
Re: Review Request 12174: Add "one-to-all" message sending strategy
Posted by Bingjing Zhang <je...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/
-----------------------------------------------------------
(Updated July 16, 2013, 11:33 p.m.)
Review request for giraph.
Changes
-------
Add new interface sendMessageToMultipleEdges(Iterator<I> vertexIdIterator, M message) to allow sending to a set of target vertex ids.
Bugs: GIRAPH-701
https://issues.apache.org/jira/browse/GIRAPH-701
Repository: giraph-git
Description
-------
Add "one-to-all" message sending strategy.
Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
To enable it, use conf.enableOneToAllMsgSending()
Diffs (updated)
-----
giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ff3f06d
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 2eeac18
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java a50f673
giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java bedaf48
giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 7ce0083
giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 21cf245
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f
giraph-core/src/main/java/org/apache/giraph/counters/GiraphStats.java 3f25508
giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java a9bf3fd
giraph-core/src/main/java/org/apache/giraph/graph/GlobalStats.java f3cbea2
giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 1d3cff0
giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac
giraph-core/src/main/java/org/apache/giraph/partition/PartitionStats.java b8eeca9
giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 26c547b
giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 52bac3f
giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java 2e60c09
giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingComputationTest.java 7346745
Diff: https://reviews.apache.org/r/12174/diff/
Testing
-------
Did
mvn clean verify
mvn clean test
Thanks,
Bingjing Zhang
Re: Review Request 12174: Add "one-to-all" message sending strategy
Posted by Bingjing Zhang <je...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/12174/
-----------------------------------------------------------
(Updated July 10, 2013, 9:12 p.m.)
Review request for giraph.
Changes
-------
Reformatted code, add metrics to count the total message bytes sent on each worker and the total message bytes sent in a superstep.
Bugs: GIRAPH-701
https://issues.apache.org/jira/browse/GIRAPH-701
Repository: giraph-git
Description
-------
Add "one-to-all" message sending strategy.
Now when sendMessageToAllEdges is invoked, Giraph may apply "one-to-all" message sending strategy.
To enable it, use conf.enableOneToAllMsgSending()
Diffs (updated)
-----
giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ff3f06d
giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 92d0926
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 2eeac18
giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java a50f673
giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 89fb3e4
giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java bedaf48
giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 7ce0083
giraph-core/src/main/java/org/apache/giraph/comm/requests/RequestType.java 4129fb8
giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerOneToAllMessagesRequest.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 74f1ba5
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c4cc96f
giraph-core/src/main/java/org/apache/giraph/counters/GiraphStats.java 3f25508
giraph-core/src/main/java/org/apache/giraph/graph/Computation.java 87d5879
giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java a9bf3fd
giraph-core/src/main/java/org/apache/giraph/graph/GlobalStats.java f3cbea2
giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 1d3cff0
giraph-core/src/main/java/org/apache/giraph/metrics/MetricNames.java cc237ac
giraph-core/src/main/java/org/apache/giraph/partition/PartitionStats.java b8eeca9
giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayOneToAllMessages.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 26c547b
giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 52bac3f
giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java 2e60c09
giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java bc5b5e2
giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingComputationTest.java 7346745
Diff: https://reviews.apache.org/r/12174/diff/
Testing
-------
Did
mvn clean verify
mvn clean test
Thanks,
Bingjing Zhang