You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by okram <gi...@git.apache.org> on 2016/05/19 11:38:09 UTC

[GitHub] incubator-tinkerpop pull request: TINKERPOP-1035: HALTED_TRAVERSER...

GitHub user okram opened a pull request:

    https://github.com/apache/incubator-tinkerpop/pull/316

    TINKERPOP-1035: HALTED_TRAVERSERS hold wrong information

    https://issues.apache.org/jira/browse/TINKERPOP-1305
    
    @dkuppitz found a severe bug in 3.2.0 in OLAP traversal chains. Interestingly enough, all the test cases we had didn't touch this problem (even though they did tangentially, but didn't test particular expectations). As such, two new tests have been added to `pageRank()` and `peerPressure()` which demonstrate the problem has been rectified.
    
    CHANGELOG
    
    ```
    * Fixed a severe bug around halted traversers in a multi-job OLAP traversal chain.
    * Ensure a separation of `GraphComputer` and `VertexProgram` configurations in `SparkGraphComputer` and `GiraphGraphComputer`.
    * `PeerPressureVertexProgram` now supports dynamic initial vote strength calculations.
    * Added `EmptyMemory` for ease of use when no memory exists.
    * Updated `VertexComputing.generateProgram()` API to include `Memory` (*breaking*).
    ```
    
    UPGRADE
    
    ```
    VertexComputing API Change
    +++++++++++++++++++++++
    
    The `VertexComputing` API is used by steps that wrap a `VertexProgram`. There is a method called `VertexComputing.generateProgram()` that has changed which now takes a second argument of `Memory`. To  upgrade, simply fix the method signature of your `VertexComputing` implementations. The `Memory` argument can be safely ignored to effect the exact same semantics as prior. However, now previous OLAP job `Memory` can be leveraged when constructing the next `VertexProgram` in an OLAP traversal chain.
    ```
    
    `mvn clean install` PASSED, integration tests PASSED, and docs built GOOD.
    
    ```
    [INFO] Apache TinkerPop .................................. SUCCESS [4.477s]
    [INFO] Apache TinkerPop :: Gremlin Shaded ................ SUCCESS [2.256s]
    [INFO] Apache TinkerPop :: Gremlin Core .................. SUCCESS [38.899s]
    [INFO] Apache TinkerPop :: Gremlin Test .................. SUCCESS [11.399s]
    [INFO] Apache TinkerPop :: Gremlin Groovy ................ SUCCESS [38.721s]
    [INFO] Apache TinkerPop :: Gremlin Groovy Test ........... SUCCESS [6.730s]
    [INFO] Apache TinkerPop :: TinkerGraph Gremlin ........... SUCCESS [3:15.097s]
    [INFO] Apache TinkerPop :: Gremlin Benchmark ............. SUCCESS [5.194s]
    [INFO] Apache TinkerPop :: Hadoop Gremlin ................ SUCCESS [10:57.415s]
    [INFO] Apache TinkerPop :: Spark Gremlin ................. SUCCESS [6:28.472s]
    [INFO] Apache TinkerPop :: Giraph Gremlin ................ SUCCESS [2:47:11.903s]
    [INFO] Apache TinkerPop :: Neo4j Gremlin ................. SUCCESS [1:08:16.166s]
    [INFO] Apache TinkerPop :: Gremlin Driver ................ SUCCESS [8.704s]
    [INFO] Apache TinkerPop :: Gremlin Server ................ SUCCESS [13:13.317s]
    [INFO] Apache TinkerPop :: Gremlin Console ............... SUCCESS [1:23.504s]
    [INFO] Apache TinkerPop :: Gremlin Archetype ............. SUCCESS [0.134s]
    [INFO] Apache TinkerPop :: Archetype - TinkerGraph ....... SUCCESS [7.257s]
    [INFO] Apache TinkerPop :: Archetype - Server ............ SUCCESS [11.883s]
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 4:33:02.004s
    [INFO] Finished at: Thu May 19 00:09:24 MDT 2016
    [INFO] Final Memory: 99M/712M
    [INFO] ------------------------------------------------------------------------
    ```
    
    VOTE +1.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/incubator-tinkerpop TINKERPOP-1305

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-tinkerpop/pull/316.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #316
    
----
commit 4ee302899da37a0ad0ebab6bc4954ed39b69d6d9
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2016-05-18T22:15:40Z

    fixed HALTED_TRAVERSER bug where traversers were not at their 'resting location'. This only shows up in multi-OLAP job chains. HALTED_TRAVERESERS (master travesal) are now propagated via Configuration and NOT via sideEffects. This is more elegant and there are now helper methods in TraversalVertexProgram to make getting master halted traversers easy. Updated the-travesal.asciidoc to reflect how to use ProgramStep with it. Had to change a method signature in VertexComputing to account for this... its slight (breaking), but no one is using this right now, I'm sure of it. And if they are, the chaneg is trivial, just add a new argument to a method signature and ignore it to make it be like how it is in 3.2.0. ImmutablePath.TailPath was not serializable and this caused issues for HALTED_TRAVERSERS as well. Everything else fixed up.

commit 07e12995ae6bed0a3b1da3942bc54371d94a66a9
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2016-05-18T23:02:22Z

    TraverserExecutor is smart to not do another iteration if the traversers will simply be returned to the master traversals as output. Updated the PageRankTest with a better test.

commit a6aacdc51eba895e04f9d75d64343470fbaca5c5
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2016-05-18T23:11:26Z

    minor tweaks to TraverserVertexProgram to make it more efficient.

commit 7d29ef32a42b143f9654457a46ab64a57446ef14
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2016-05-18T23:12:40Z

    minor tweaks to TraverserVertexProgram to make it more efficient.

commit 6107ff95e47129d6266a68410c01c17b04af95cf
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2016-05-19T01:24:57Z

    made a PageRank test a bit more robust. running full integration over night.

commit b100f033e02ee5647dd31b2d556a1dcadac29818
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2016-05-19T01:36:18Z

    added EmptyMemory and its used as the memory for the first generateProgram() of an OLAP chain. Much better than Optional(Memory).

commit 5bcf0b01c7bfc77e9eea8fdb7b2c4c54a4fc0000
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2016-05-19T11:23:36Z

    added traversal-based vote strength test to PeerPressureTest. Cleaned up internal classes of TraversalVertexProgram a bit for better organization. Generalized Phase enum so it can be used by other computer-based classes. added comments to TraveralVertexProgram.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop pull request: TINKERPOP-1035: HALTED_TRAVERSER...

Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:

    https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220387063
  
    Can you rename this PR? You transposed the number and the wrong JIRA ticket is getting the updates.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-tinkerpop/pull/316


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...

Posted by okram <gi...@git.apache.org>.
Github user okram commented on the pull request:

    https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220440890
  
    These are "master traversal" traversers. Even before when it was stored in sideeffects, those are ultimately stored in `Configuration` (`TRAVERSAL`). Its the same scalability as before, just better semantics as we shouldn't store halted traversers as side-effects (they have "left the traversal"). Finally, since the beginning, OLAP reductions to the master traversal are only as scalable as the single machine it comes back to. Thus, groups(), limits(), groupCounts(), sums(), etc. are important. There is no `g.V()` on a billion vertex graph and stream back results.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...

Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the pull request:

    https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220732937
  
    Dozens of manual tests later...
    
    VOTE: +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop pull request: TINKERPOP-1035: HALTED_TRAVERSER...

Posted by okram <gi...@git.apache.org>.
Github user okram commented on the pull request:

    https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220384591
  
    This also solves https://issues.apache.org/jira/browse/TINKERPOP-1298


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...

Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the pull request:

    https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220427562
  
    VOTE +1 - `$ docker/build.sh -t -i -n` is all good


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-tinkerpop pull request: TINKERPOP-1305: HALTED_TRAVERSER...

Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the pull request:

    https://github.com/apache/incubator-tinkerpop/pull/316#issuecomment-220434331
  
    Even if this PR gets another +1, please don't merge yet, as I'm still doing lots of manual tests these days. I'm still not sure if I like halted traversers being stored in the config (simply because I'm not yet at the point where I've tested this particular change).
    
    And even if my tests are going to run smoothly, I'm still worried about how scalable this approach is. In a distributed environment, how is the configuration distributed between worker iterations? Where is it stored? Will it be (re)loaded for each worker iteration? If so, how long does it take to load a config that holds millions of halted traversers?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---