You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Jan van der Lugt (JIRA)" <ji...@apache.org> on 2012/05/22 00:39:41 UTC
[jira] [Created] (GIRAPH-192) Move aggregators to a seperate
sub-package
Jan van der Lugt created GIRAPH-192:
---------------------------------------
Summary: Move aggregators to a seperate sub-package
Key: GIRAPH-192
URL: https://issues.apache.org/jira/browse/GIRAPH-192
Project: Giraph
Issue Type: Improvement
Components: examples
Reporter: Jan van der Lugt
Priority: Minor
Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
for int/long/float/double: min, max, overwrite
for long/double: product, sum
for boolean: and, or, overwrite
Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396424#comment-13396424 ]
Jan van der Lugt commented on GIRAPH-192:
-----------------------------------------
Also, I included average-aggregators for doubles and floats. Don't know why I did not include those in the first place.
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan van der Lugt updated GIRAPH-192:
------------------------------------
Attachment: (was: GIRAPH-192.patch)
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398531#comment-13398531 ]
Hudson commented on GIRAPH-192:
-------------------------------
Integrated in giraph-trunk-non-secure #8 (See [https://builds.apache.org/job/giraph-trunk-non-secure/8/])
GIRAPH-192. Move aggregators to a separate sub-package. Contributed by Jan van der Lugt. (Revision 1352585)
Result = SUCCESS
jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352585
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanAndAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOrAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/package-info.java
* /giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/VerifyMessage.java
* /giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestBooleanAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestDoubleAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestFloatAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestIntAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestLongAggregators.java
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan van der Lugt updated GIRAPH-192:
------------------------------------
Attachment: GIRAPH-192.patch
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Reporter: Jan van der Lugt
> Priority: Minor
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, overwrite
> for long/double: product, sum
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398564#comment-13398564 ]
Hudson commented on GIRAPH-192:
-------------------------------
Integrated in giraph-trunk-hadoop-1.0 #7 (See [https://builds.apache.org/job/giraph-trunk-hadoop-1.0/7/])
GIRAPH-192. Move aggregators to a separate sub-package. Contributed by Jan van der Lugt. (Revision 1352585)
Result = SUCCESS
jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352585
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanAndAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOrAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/package-info.java
* /giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/VerifyMessage.java
* /giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestBooleanAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestDoubleAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestFloatAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestIntAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestLongAggregators.java
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398558#comment-13398558 ]
Hudson commented on GIRAPH-192:
-------------------------------
Integrated in giraph-trunk-hadoop-0.20.203 #9 (See [https://builds.apache.org/job/giraph-trunk-hadoop-0.20.203/9/])
GIRAPH-192. Move aggregators to a separate sub-package. Contributed by Jan van der Lugt. (Revision 1352585)
Result = SUCCESS
jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352585
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanAndAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOrAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/package-info.java
* /giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/VerifyMessage.java
* /giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestBooleanAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestDoubleAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestFloatAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestIntAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestLongAggregators.java
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396937#comment-13396937 ]
Jakob Homan commented on GIRAPH-192:
------------------------------------
(also, it's not necessary to delete the previous version of the patch when attaching a new one. JIRA will go ahead and grey out the older versions and it's good to keep them around to avoid confusion.)
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397650#comment-13397650 ]
Jan van der Lugt commented on GIRAPH-192:
-----------------------------------------
Nope, it seems I made some mistakes :-) Only fixed the ones I saw on the console, didn't know there was an html file too. Should be better now, learning something every day.
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan van der Lugt updated GIRAPH-192:
------------------------------------
Attachment: GIRAPH-192.patch
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396304#comment-13396304 ]
Jakob Homan commented on GIRAPH-192:
------------------------------------
Hey Jan. Had a look at the patch. A few questions:
* What's the use case for the overwrite aggregator? It's nondeterministic in operation (particularly in a distributed environment), so I'm having trouble seeing why it's needed?
* Rather than separate Int/Long and Float/Double aggregators, perhaps we can just have long and float, respectively, and rely on the greater size/precision? Each aggregator will take twice as much memory, but there should be relatively few of them so this may not be so bad. Thoughts?
* It seems like it should be possible to parameterize these classes on pairs of both the Java type and corresponding Writable version (ie <Long, LongWritable>). Not great, but might lead to less code. Is this viable?
* The new aggregators will require unit tests and we may as well take the opportunity to add unit tests for the existing ones (if not, at least add a newbie JIRA for adding them).
* What happens when we overflow on, for instance, product aggregator with an int? This should be tested as well.
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398539#comment-13398539 ]
Hudson commented on GIRAPH-192:
-------------------------------
Integrated in giraph-trunk-hadoop-0.23 #8 (See [https://builds.apache.org/job/giraph-trunk-hadoop-0.23/8/])
GIRAPH-192. Move aggregators to a separate sub-package. Contributed by Jan van der Lugt. (Revision 1352585)
Result = SUCCESS
jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352585
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanAndAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOrAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/package-info.java
* /giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/VerifyMessage.java
* /giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestBooleanAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestDoubleAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestFloatAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestIntAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestLongAggregators.java
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Benjamin Heitmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280909#comment-13280909 ]
Benjamin Heitmann commented on GIRAPH-192:
------------------------------------------
Jan, your description of this issue sounds like you already used aggregators extensively.
I have recently tried using aggregators for generating statistics about my algorithm,
and I am not completely sure if the reported numbers are accurate.
One issue which I had, is that aggregators which get a reset e.g. between supersteps, do not report accurate numbers when manually compared to aggregators which count the same thing, but which did not get a reset.
The other issue, is that some aggregators (without a reset) reported way to high overall numbers.
The last issue might really be due to wrong code on my side. But the first issue really seemed like a bug to me.
Jan, do have an insight into the inner workings of the aggregators? Are there unit tests for the aggregators? Is it possible that they are buggy in some way (similar to what I described), or would you certain that their code is reliable, and that they just need to be moved to their own sub-package ?
It would be really good to have aggregators working properly outside of the example package, not just for me, but for everybody, as they form a part of the original Google Pregel BSP design.
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Reporter: Jan van der Lugt
> Priority: Minor
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281095#comment-13281095 ]
Jan van der Lugt commented on GIRAPH-192:
-----------------------------------------
The aggregators I provided in the patch are the same as the ones in the .example package, just slightly modified by replacing the types or operations (generic aggregators are sadly not a possibility). The product aggregators, for example, are the same as the sum aggregators, except with += replaced by *=. You are right in mentioning that aggregators are part of the Google Pregel BSP design, but so are combiners. The problem with combiners is that they are dependent on specific message formats, while aggregators work directly on values. This is why I believe Giraph should provide reusable aggregators, but not combiners.
Regarding your first issue with aggregators: I implemented Pagerank with an aggregator that is used to calculate the global sum of errors. The values calculated by this aggregator were exactly the same as calculated by another algorithm (built on a different, in-memory, single machine graph framework). This suggests to me that they are, at least in my case, working correctly. You might want to check your code again, is the reset performed in the preSuperstep() function? The mailing list might be a better place to discuss this than this JIRA, however.
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Reporter: Jan van der Lugt
> Priority: Minor
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan van der Lugt updated GIRAPH-192:
------------------------------------
Attachment: (was: GIRAPH-192.patch)
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Reporter: Jan van der Lugt
> Priority: Minor
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, overwrite
> for long/double: product, sum
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398567#comment-13398567 ]
Hudson commented on GIRAPH-192:
-------------------------------
Integrated in giraph-formats-contrib #9 (See [https://builds.apache.org/job/giraph-formats-contrib/9/])
GIRAPH-192. Move aggregators to a separate sub-package. Contributed by Jan van der Lugt. (Revision 1352585)
Result = SUCCESS
jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352585
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanAndAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOrAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/package-info.java
* /giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/VerifyMessage.java
* /giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestBooleanAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestDoubleAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestFloatAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestIntAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestLongAggregators.java
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan van der Lugt updated GIRAPH-192:
------------------------------------
Attachment: GIRAPH-192.patch
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Reporter: Jan van der Lugt
> Priority: Minor
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, overwrite
> for long/double: product, sum
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated GIRAPH-192:
-------------------------------
Affects Version/s: 0.2.0
Fix Version/s: 0.2.0
Assignee: Jan van der Lugt
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396810#comment-13396810 ]
Jakob Homan commented on GIRAPH-192:
------------------------------------
OK, this all sounds reasonable. We're being severely hobbled by a combination of Java's type system and the Writable serialization format, leading to lots of boilerplate code. But I don't see any easy way out of that right now.
The patch has a bunch of checkstyle errors (run mvn verify to get the list in checkstyle-results.xml). After those are corrected and unit tests added, I think we're good to go.
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398514#comment-13398514 ]
Jan van der Lugt commented on GIRAPH-192:
-----------------------------------------
Glad to be able to contribute. Thanks for your help!
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398541#comment-13398541 ]
Hudson commented on GIRAPH-192:
-------------------------------
Integrated in Giraph-trunk-Commit #120 (See [https://builds.apache.org/job/Giraph-trunk-Commit/120/])
GIRAPH-192. Move aggregators to a separate sub-package. Contributed by Jan van der Lugt. (Revision 1352585)
Result = SUCCESS
jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352585
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanAndAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOrAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/package-info.java
* /giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/VerifyMessage.java
* /giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestBooleanAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestDoubleAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestFloatAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestIntAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestLongAggregators.java
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan van der Lugt updated GIRAPH-192:
------------------------------------
Attachment: GIRAPH-192.patch
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397046#comment-13397046 ]
Jakob Homan commented on GIRAPH-192:
------------------------------------
Looking good except it doesn't appear that the *AverageAggregators made it into the latest patch. Unit tests look great. (there's an assert{True|False} if you want to make some of the assertEquals simpler, but it's a matter of taste and fine either way).
Assuming an overflow happens (which I think is a distinct possibility), it may be best to just let it happen. I imagine it'll be a relatively common failure, so it may be worth catching and failing gracefully, but that can be done in another issue.
Except for the missing classes, this is ready to go.
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan van der Lugt updated GIRAPH-192:
------------------------------------
Description:
Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
for int/long/float/double: min, max, product, sum, overwrite
for boolean: and, or, overwrite
Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
was:
Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
for int/long/float/double: min, max, overwrite
for long/double: product, sum
for boolean: and, or, overwrite
Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
Remaining Estimate: 2h (was: 3h)
Original Estimate: 2h (was: 3h)
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Reporter: Jan van der Lugt
> Priority: Minor
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan resolved GIRAPH-192.
--------------------------------
Resolution: Fixed
No problem. Thanks for the quick turn-around. +1. I've committed this. Resolving as fixed. Thanks for the contribution, Jan!
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397074#comment-13397074 ]
Jakob Homan commented on GIRAPH-192:
------------------------------------
I'm getting 66 checkstyle errors now. Did those fixes not get merged in?
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398574#comment-13398574 ]
Hudson commented on GIRAPH-192:
-------------------------------
Integrated in giraph-trunk-hadoop-trunk #7 (See [https://builds.apache.org/job/giraph-trunk-hadoop-trunk/7/])
GIRAPH-192. Move aggregators to a separate sub-package. Contributed by Jan van der Lugt. (Revision 1352585)
Result = FAILURE
jghoman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352585
Files :
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanAndAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOrAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/BooleanOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/DoubleSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatAverageAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/FloatSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/IntSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongMinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongOverwriteAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongProductAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/aggregators/package-info.java
* /giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/LongSumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MaxAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/MinAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/SumAggregator.java
* /giraph/trunk/src/main/java/org/apache/giraph/examples/VerifyMessage.java
* /giraph/trunk/src/test/java/org/apache/giraph/TestBspBasic.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestBooleanAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestDoubleAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestFloatAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestIntAggregators.java
* /giraph/trunk/src/test/java/org/apache/giraph/aggregators/TestLongAggregators.java
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a separate
sub-package
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated GIRAPH-192:
-------------------------------
Summary: Move aggregators to a separate sub-package (was: Move aggregators to a seperate sub-package)
> Move aggregators to a separate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan van der Lugt updated GIRAPH-192:
------------------------------------
Attachment: GIRAPH-192.patch
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan van der Lugt updated GIRAPH-192:
------------------------------------
Attachment: GIRAPH-192.patch
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396415#comment-13396415 ]
Jan van der Lugt commented on GIRAPH-192:
-----------------------------------------
Hi Jakob,
- The overwrite aggregator is only useful when using master.compute(). The master sets a value in the aggregator, overwriting the previous value (for example some global state the algorithm is in) and the workers read this value. I've changed the class description for these classes.
- I thought about having just long and double aggregators (with helper methods for settings ints and floats), but my conclusion is that when a user wants to depend on floats, for example to reproduce the results of an algorithm that uses floats, he should be able to. Of course it's usually better to use longs/doubles (especially when you are summing or calculating an average), but I did not want to make the choice for the user. I don't think it's a big problem to have them both and let the user decide, but if you think it's better to remove them, I'm ok with that.
- I thought about (and tried) making generic aggregators, but this is difficult for a few different reasons. Most importantly, the .get() method is not defined in Writable, only in for example LongWritable. There is no common superinterface for LongWritable, IntWritable, FloatWritable and DoubleWritable except for Writable, which is not usable because of the reason mentioned before.
- I will write the tests or file a JIRA for this once the shape and form of the aggregators ia agreed on, is that ok?
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-192) Move aggregators to a seperate
sub-package
Posted by "Jan van der Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397009#comment-13397009 ]
Jan van der Lugt commented on GIRAPH-192:
-----------------------------------------
Yes, Java's type system and Hadoop's Writables are not playing nice together. I've changed several things:
- deleted the old aggregators (was not included in previous patch)
- added unit tests. not really sure if this is ok, haven't written a lot of unit tests before
- boolean and / or aggregators had the opposite operators in them
- product aggregators start with a default value of 1 (found this bug with the unit tests, yay!)
- fixed checkstyle warnings
Don't know how we would test the overflow behavior in a unit test, do you have ideas for this, Jakob?
> Move aggregators to a seperate sub-package
> ------------------------------------------
>
> Key: GIRAPH-192
> URL: https://issues.apache.org/jira/browse/GIRAPH-192
> Project: Giraph
> Issue Type: Improvement
> Components: examples
> Affects Versions: 0.2.0
> Reporter: Jan van der Lugt
> Assignee: Jan van der Lugt
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: GIRAPH-192.patch, GIRAPH-192.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Since aggregators will be re-used throughout many projects and algorithms, it makes sense to implement the most common ones in a separate sub-package. This will reduce the time required for users when they implement their projects based on Giraph, because the required aggregators are already in place. I implemented the following ones:
> for int/long/float/double: min, max, product, sum, overwrite
> for boolean: and, or, overwrite
> Most of them speak for themselves, except for the overwrite one. This aggregator simply overwrites the stored value when a new value is aggregated. This is useful when one node is in some way a master node (for example a source node in an routing algorithm), and this node wants to broadcast a value to all other nodes.
> Attached is a patch against trunk implementing the aggregators and patching some existing files so they use the .aggregators package instead of the .examples one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira