You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Maja Kabiljo <ma...@fb.com> on 2013/04/10 02:47:46 UTC
Review Request: GIRAPH-615: Add support for multithreaded output
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10389/
-----------------------------------------------------------
Review request for giraph.
Description
-------
Some output formats, like giraph-hive stuff, can write output in parallel. We should provide an option to choose number of threads to use for storing output.
While there, I cleaned up some stuff around ExecutorService calls.
This addresses bug GIRAPH-615.
https://issues.apache.org/jira/browse/GIRAPH-615
Diffs
-----
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 90b05e3
giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 95c9862
giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java 01a67dd
giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java 51ed4f6
giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java abca4c4
giraph-core/src/main/java/org/apache/giraph/utils/CallableFactory.java PRE-CREATION
giraph-core/src/main/java/org/apache/giraph/utils/ProgressableUtils.java 77eb49a
giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java c20d06e
giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallableFactory.java 9297ac1
giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallableFactory.java aebca81
Diff: https://reviews.apache.org/r/10389/diff/
Testing
-------
mvn clean install
Tested on real application with all options (set number of output threads with an without thread safe output), using giraph-hive.
Thanks,
Maja Kabiljo
Re: Review Request: GIRAPH-615: Add support for multithreaded output
Posted by Maja Kabiljo <ma...@fb.com>.
> On April 10, 2013, 5:58 p.m., Avery Ching wrote:
> > This is a nice refactor. Also a few hidden improvements such as not putting the partition back. Looks good!
Thanks Avery, committing!
- Maja
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10389/#review18960
-----------------------------------------------------------
On April 10, 2013, 12:47 a.m., Maja Kabiljo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10389/
> -----------------------------------------------------------
>
> (Updated April 10, 2013, 12:47 a.m.)
>
>
> Review request for giraph.
>
>
> Description
> -------
>
> Some output formats, like giraph-hive stuff, can write output in parallel. We should provide an option to choose number of threads to use for storing output.
>
> While there, I cleaned up some stuff around ExecutorService calls.
>
>
> This addresses bug GIRAPH-615.
> https://issues.apache.org/jira/browse/GIRAPH-615
>
>
> Diffs
> -----
>
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 90b05e3
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 95c9862
> giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java 01a67dd
> giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java 51ed4f6
> giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java abca4c4
> giraph-core/src/main/java/org/apache/giraph/utils/CallableFactory.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/utils/ProgressableUtils.java 77eb49a
> giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java c20d06e
> giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallableFactory.java 9297ac1
> giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallableFactory.java aebca81
>
> Diff: https://reviews.apache.org/r/10389/diff/
>
>
> Testing
> -------
>
> mvn clean install
> Tested on real application with all options (set number of output threads with an without thread safe output), using giraph-hive.
>
>
> Thanks,
>
> Maja Kabiljo
>
>
Re: Review Request: GIRAPH-615: Add support for multithreaded output
Posted by Avery Ching <av...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10389/#review18960
-----------------------------------------------------------
Ship it!
This is a nice refactor. Also a few hidden improvements such as not putting the partition back. Looks good!
- Avery Ching
On April 10, 2013, 12:47 a.m., Maja Kabiljo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10389/
> -----------------------------------------------------------
>
> (Updated April 10, 2013, 12:47 a.m.)
>
>
> Review request for giraph.
>
>
> Description
> -------
>
> Some output formats, like giraph-hive stuff, can write output in parallel. We should provide an option to choose number of threads to use for storing output.
>
> While there, I cleaned up some stuff around ExecutorService calls.
>
>
> This addresses bug GIRAPH-615.
> https://issues.apache.org/jira/browse/GIRAPH-615
>
>
> Diffs
> -----
>
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 90b05e3
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 95c9862
> giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java 01a67dd
> giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java 51ed4f6
> giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java abca4c4
> giraph-core/src/main/java/org/apache/giraph/utils/CallableFactory.java PRE-CREATION
> giraph-core/src/main/java/org/apache/giraph/utils/ProgressableUtils.java 77eb49a
> giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java c20d06e
> giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallableFactory.java 9297ac1
> giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallableFactory.java aebca81
>
> Diff: https://reviews.apache.org/r/10389/diff/
>
>
> Testing
> -------
>
> mvn clean install
> Tested on real application with all options (set number of output threads with an without thread safe output), using giraph-hive.
>
>
> Thanks,
>
> Maja Kabiljo
>
>