You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by okram <gi...@git.apache.org> on 2016/03/30 22:26:38 UTC
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1225: Do a "rolling re...
GitHub user okram opened a pull request:
https://github.com/apache/incubator-tinkerpop/pull/285
TINKERPOP-1225: Do a "rolling reduce" for GroupXXXStep in OLAP.
https://issues.apache.org/jira/browse/TINKERPOP-1225
We now can do mid-barrier reductions in with `group()` on OLAP. This is huge as this means that if you have an reducer in your `by()`-valueTraversal, the stream is constantly reduced to limit memory consumption. Its more expensive in terms of time (not by much) for small data, but for large data, no worries about OME with group() (both both OLTP and OLAP).
CHANGELOG
```
* `GroupStep` and `GroupSideEffectStep` make use of mid-traversal reducers to limit memory consumption in OLAP.
```
VOTE +1
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-tinkerpop TINKERPOP-1225
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tinkerpop/pull/285.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #285
----
commit 8eb53815acfcec05316dcf39bd91d9e3a43d971a
Author: Marko A. Rodriguez <ok...@gmail.com>
Date: 2016-03-30T20:24:38Z
GroupStep and GroupSideEffectStep now make use mid-traversal barriers to do data reduction on the fly in order to limite the memory footprint and reduce the chances of OME. OLTP always did this, but now OLAP (which needs it more) does it. This is epic. Also fixed a minor bug in ReducingBarrierStep. Added some more GroupTest test cases -- one that does a groupCount() instead of a group() just to make sure things are working as expected.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1225: Do a "rolling re...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-tinkerpop/pull/285
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1225: Do a "rolling re...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/285#issuecomment-203986734
VOTE +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1225: Do a "rolling re...
Posted by twilmes <gi...@git.apache.org>.
Github user twilmes commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/285#issuecomment-204087712
VOTE: +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1225: Do a "rolling re...
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/285#issuecomment-204089672
VOTE: +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1225: Do a "rolling re...
Posted by okram <gi...@git.apache.org>.
Github user okram commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/285#issuecomment-203927074
```
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache TinkerPop .................................. SUCCESS [4.350s]
[INFO] Apache TinkerPop :: Gremlin Shaded ................ SUCCESS [2.302s]
[INFO] Apache TinkerPop :: Gremlin Core .................. SUCCESS [40.323s]
[INFO] Apache TinkerPop :: Gremlin Test .................. SUCCESS [12.404s]
[INFO] Apache TinkerPop :: Gremlin Groovy ................ SUCCESS [42.373s]
[INFO] Apache TinkerPop :: Gremlin Groovy Test ........... SUCCESS [6.747s]
[INFO] Apache TinkerPop :: TinkerGraph Gremlin ........... SUCCESS [3:24.962s]
[INFO] Apache TinkerPop :: Gremlin Benchmark ............. SUCCESS [10.390s]
[INFO] Apache TinkerPop :: Hadoop Gremlin ................ SUCCESS [11:27.951s]
[INFO] Apache TinkerPop :: Spark Gremlin ................. SUCCESS [7:01.345s]
[INFO] Apache TinkerPop :: Giraph Gremlin ................ SUCCESS [2:41:47.692s]
[INFO] Apache TinkerPop :: Neo4j Gremlin ................. SUCCESS [17:53.279s]
[INFO] Apache TinkerPop :: Gremlin Driver ................ SUCCESS [10.358s]
[INFO] Apache TinkerPop :: Gremlin Server ................ SUCCESS [12:52.175s]
[INFO] Apache TinkerPop :: Gremlin Console ............... SUCCESS [1:25.610s]
[INFO] Apache TinkerPop :: Gremlin Archetype ............. SUCCESS [0.086s]
[INFO] Apache TinkerPop :: Archetype - TinkerGraph ....... SUCCESS [4.702s]
[INFO] Apache TinkerPop :: Archetype - Server ............ SUCCESS [10.319s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:38:17.858s
[INFO] Finished at: Wed Mar 30 22:57:45 MDT 2016
[INFO] Final Memory: 99M/765M
[INFO] ------------------------------------------------------------------------
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---