You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by okram <gi...@git.apache.org> on 2016/05/19 11:38:09 UTC
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1035: HALTED_TRAVERSER...
GitHub user okram opened a pull request:
https://github.com/apache/incubator-tinkerpop/pull/316
TINKERPOP-1035: HALTED_TRAVERSERS hold wrong information
https://issues.apache.org/jira/browse/TINKERPOP-1305
@dkuppitz found a severe bug in 3.2.0 in OLAP traversal chains. Interestingly enough, all the test cases we had didn't touch this problem (even though they did tangentially, but didn't test particular expectations). As such, two new tests have been added to `pageRank()` and `peerPressure()` which demonstrate the problem has been rectified.
CHANGELOG
```
* Fixed a severe bug around halted traversers in a multi-job OLAP traversal chain.
* Ensure a separation of `GraphComputer` and `VertexProgram` configurations in `SparkGraphComputer` and `GiraphGraphComputer`.
* `PeerPressureVertexProgram` now supports dynamic initial vote strength calculations.
* Added `EmptyMemory` for ease of use when no memory exists.
* Updated `VertexComputing.generateProgram()` API to include `Memory` (*breaking*).
```
UPGRADE
```
VertexComputing API Change
+++++++++++++++++++++++
The `VertexComputing` API is used by steps that wrap a `VertexProgram`. There is a method called `VertexComputing.generateProgram()` that has changed which now takes a second argument of `Memory`. To upgrade, simply fix the method signature of your `VertexComputing` implementations. The `Memory` argument can be safely ignored to effect the exact same semantics as prior. However, now previous OLAP job `Memory` can be leveraged when constructing the next `VertexProgram` in an OLAP traversal chain.
```
`mvn clean install` PASSED, integration tests PASSED, and docs built GOOD.
```
[INFO] Apache TinkerPop .................................. SUCCESS [4.477s]
[INFO] Apache TinkerPop :: Gremlin Shaded ................ SUCCESS [2.256s]
[INFO] Apache TinkerPop :: Gremlin Core .................. SUCCESS [38.899s]
[INFO] Apache TinkerPop :: Gremlin Test .................. SUCCESS [11.399s]
[INFO] Apache TinkerPop :: Gremlin Groovy ................ SUCCESS [38.721s]
[INFO] Apache TinkerPop :: Gremlin Groovy Test ........... SUCCESS [6.730s]
[INFO] Apache TinkerPop :: TinkerGraph Gremlin ........... SUCCESS [3:15.097s]
[INFO] Apache TinkerPop :: Gremlin Benchmark ............. SUCCESS [5.194s]
[INFO] Apache TinkerPop :: Hadoop Gremlin ................ SUCCESS [10:57.415s]
[INFO] Apache TinkerPop :: Spark Gremlin ................. SUCCESS [6:28.472s]
[INFO] Apache TinkerPop :: Giraph Gremlin ................ SUCCESS [2:47:11.903s]
[INFO] Apache TinkerPop :: Neo4j Gremlin ................. SUCCESS [1:08:16.166s]
[INFO] Apache TinkerPop :: Gremlin Driver ................ SUCCESS [8.704s]
[INFO] Apache TinkerPop :: Gremlin Server ................ SUCCESS [13:13.317s]
[INFO] Apache TinkerPop :: Gremlin Console ............... SUCCESS [1:23.504s]
[INFO] Apache TinkerPop :: Gremlin Archetype ............. SUCCESS [0.134s]
[INFO] Apache TinkerPop :: Archetype - TinkerGraph ....... SUCCESS [7.257s]
[INFO] Apache TinkerPop :: Archetype - Server ............ SUCCESS [11.883s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4:33:02.004s
[INFO] Finished at: Thu May 19 00:09:24 MDT 2016
[INFO] Final Memory: 99M/712M
[INFO] ------------------------------------------------------------------------
```
VOTE +1.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-tinkerpop TINKERPOP-1305
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-tinkerpop/pull/316.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #316
----
commit 4ee302899da37a0ad0ebab6bc4954ed39b69d6d9
Author: Marko A. Rodriguez <ok...@gmail.com>
Date: 2016-05-18T22:15:40Z
fixed HALTED_TRAVERSER bug where traversers were not at their 'resting location'. This only shows up in multi-OLAP job chains. HALTED_TRAVERESERS (master travesal) are now propagated via Configuration and NOT via sideEffects. This is more elegant and there are now helper methods in TraversalVertexProgram to make getting master halted traversers easy. Updated the-travesal.asciidoc to reflect how to use ProgramStep with it. Had to change a method signature in VertexComputing to account for this... its slight (breaking), but no one is using this right now, I'm sure of it. And if they are, the chaneg is trivial, just add a new argument to a method signature and ignore it to make it be like how it is in 3.2.0. ImmutablePath.TailPath was not serializable and this caused issues for HALTED_TRAVERSERS as well. Everything else fixed up.
commit 07e12995ae6bed0a3b1da3942bc54371d94a66a9
Author: Marko A. Rodriguez <ok...@gmail.com>
Date: 2016-05-18T23:02:22Z
TraverserExecutor is smart to not do another iteration if the traversers will simply be returned to the master traversals as output. Updated the PageRankTest with a better test.
commit a6aacdc51eba895e04f9d75d64343470fbaca5c5
Author: Marko A. Rodriguez <ok...@gmail.com>
Date: 2016-05-18T23:11:26Z
minor tweaks to TraverserVertexProgram to make it more efficient.
commit 7d29ef32a42b143f9654457a46ab64a57446ef14
Author: Marko A. Rodriguez <ok...@gmail.com>
Date: 2016-05-18T23:12:40Z
minor tweaks to TraverserVertexProgram to make it more efficient.
commit 6107ff95e47129d6266a68410c01c17b04af95cf
Author: Marko A. Rodriguez <ok...@gmail.com>
Date: 2016-05-19T01:24:57Z
made a PageRank test a bit more robust. running full integration over night.
commit b100f033e02ee5647dd31b2d556a1dcadac29818
Author: Marko A. Rodriguez <ok...@gmail.com>
Date: 2016-05-19T01:36:18Z
added EmptyMemory and its used as the memory for the first generateProgram() of an OLAP chain. Much better than Optional(Memory).
commit 5bcf0b01c7bfc77e9eea8fdb7b2c4c54a4fc0000
Author: Marko A. Rodriguez <ok...@gmail.com>
Date: 2016-05-19T11:23:36Z
added traversal-based vote strength test to PeerPressureTest. Cleaned up internal classes of TraversalVertexProgram a bit for better organization. Generalized Phase enum so it can be used by other computer-based classes. added comments to TraveralVertexProgram.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1035: HALTED_TRAVERSER...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220387063
Can you rename this PR? You transposed the number and the wrong JIRA ticket is getting the updates.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-tinkerpop/pull/316
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...
Posted by okram <gi...@git.apache.org>.
Github user okram commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220440890
These are "master traversal" traversers. Even before when it was stored in sideeffects, those are ultimately stored in `Configuration` (`TRAVERSAL`). Its the same scalability as before, just better semantics as we shouldn't store halted traversers as side-effects (they have "left the traversal"). Finally, since the beginning, OLAP reductions to the master traversal are only as scalable as the single machine it comes back to. Thus, groups(), limits(), groupCounts(), sums(), etc. are important. There is no `g.V()` on a billion vertex graph and stream back results.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220732937
Dozens of manual tests later...
VOTE: +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1035: HALTED_TRAVERSER...
Posted by okram <gi...@git.apache.org>.
Github user okram commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220384591
This also solves https://issues.apache.org/jira/browse/TINKERPOP-1298
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220427562
VOTE +1 - `$ docker/build.sh -t -i -n` is all good
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220434331
Even if this PR gets another +1, please don't merge yet, as I'm still doing lots of manual tests these days. I'm still not sure if I like halted traversers being stored in the config (simply because I'm not yet at the point where I've tested this particular change).
And even if my tests are going to run smoothly, I'm still worried about how scalable this approach is. In a distributed environment, how is the configuration distributed between worker iterations? Where is it stored? Will it be (re)loaded for each worker iteration? If so, how long does it take to load a config that holds millions of halted traversers?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---