You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by okram <gi...@git.apache.org> on 2017/01/18 18:17:57 UTC

[GitHub] tinkerpop pull request #538: TINKERPOP-1248: OrderGlobalStep should use loca...

GitHub user okram opened a pull request:

    https://github.com/apache/tinkerpop/pull/538

    TINKERPOP-1248: OrderGlobalStep should use local star graph to compute sorts, prior to reduction.

    https://issues.apache.org/jira/browse/TINKERPOP-1248
    
    We now support having the projected data of both `OrderGlobalStep` and `SampleGlobalStep` being co-located with the `Traverser` via a traverser wrapper called `ProjectedTraverser`. This allows us to order traversers based on any projects of the local star graph -- NOT just `PathProcessor.ID`. This also reduces the complexity of ordering with `by()`-modulation as we don't have to constantly apply a projection traversal. Finally, this reduces the chances of problems with serializing the order/sample traversals -- e.g. what if side-effects are accessed and the traversals in the bi-operators are detached from the parent traversal.
    
    --- before I VOTE +1, I'm going to see if I can use this same model for `GroupStep`. If so, huge. If its complicated, will push off for the next tp32/ release.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/tinkerpop TINKERPOP-1248

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/tinkerpop/pull/538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #538
    
----
commit 501e97a1ecb23f76b2fddba8eaed1dba4a5a839e
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2017-01-18T16:08:24Z

    Created OrdertedTraverser which allows us to move beyond the star graph for OrderGlobalStep by()-projections. Moreover, it reduces the complexity of ordering as the objects of comparison are already determined. Going to generalize fully to a ProjectedTraverser which will allow us to do the same for SampleGlobalStep, DedupGlobalStep, and down the road maintain order even as the computation is re-distributed to workers.

commit 5045f67f469e163d1363f953672a3f38b4ff2a3f
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2017-01-18T16:38:21Z

    We now have ProjectedTraveser which is a Traverser with List<Object> of projections. OrderGlobalStep uses this and thus, can order for everything within the local star graph. Added MultiComparator which is like ChainedComparator but doesn't contain traversal projections -- just comparators.

commit b2f0c57df6fd9191904213622ae718a0790d7a03
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2017-01-18T18:07:32Z

    Removed a PathProcessor.ID constraint from ComputerVerificationStrategy. Moreover, sampling and ordering is more efficient as the projected data is co-located with the traverser in the new ProjectedTraverser wrapper. Going to leave it at this for tp32/... Moving forward, we can make it so we don't need to DetachFactory.detach(true) for CollectingBarrierStep by maintaining 'future data.' Its complicated and I don't want to introduce potential bugs.

commit 91e1f50c8b95d295a86cba6f3c9db7a002664233
Author: Marko A. Rodriguez <ok...@gmail.com>
Date:   2017-01-18T18:17:05Z

    updated CHANGELOG and tweaked ProjectTraverser variable naming.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tinkerpop issue #538: TINKERPOP-1248: OrderGlobalStep should use local star ...

Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:

    https://github.com/apache/tinkerpop/pull/538
  
    Merged #539 and #538 together locally and all tests pass with `docker/build.sh -t -n -i`
    
    VOTE +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tinkerpop issue #538: TINKERPOP-1248: OrderGlobalStep should use local star ...

Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:

    https://github.com/apache/tinkerpop/pull/538
  
    I see that you added new Gryo registrations. I assume that GraphSON can serialize `ProjectedTraverser` as there is a general serializer for that. What about `MultiComparator` - is that a concern? To extend on this issue given my recent work in the testing of IO, I'm wondering why certain registrations in Gryo don't require equal "registrations" in GraphSON. Are our tests coming up short in ensuring they have parity? Are we unnecessarily registering in Gryo sometimes? 
    
    I noticed there's no additional tests. I assume that this is because existing tests cover this functionality and no new breaks equals no problems.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tinkerpop issue #538: TINKERPOP-1248: OrderGlobalStep should use local star ...

Posted by okram <gi...@git.apache.org>.
Github user okram commented on the issue:

    https://github.com/apache/tinkerpop/pull/538
  
    ```
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 03:06 h
    [INFO] Finished at: 2017-01-18T19:11:37-07:00
    [INFO] Final Memory: 136M/1242M
    [INFO] ------------------------------------------------------------------------
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tinkerpop issue #538: TINKERPOP-1248: OrderGlobalStep should use local star ...

Posted by okram <gi...@git.apache.org>.
Github user okram commented on the issue:

    https://github.com/apache/tinkerpop/pull/538
  
    So `MultiComparator` is internal to a `Traversal`. The way in which GraphSON serializes traversals is via their Bytecode. Thus, when the bytecode gets recompiled on the deserializing end, the `MultiComparator` is recreated. The reason this is needed in Gryo is because we use Gryo as the binary format internal to Java-based systems. Giraph and Spark register the `MultiComparator` in their Aggregator framework and Gryo is the serializer for it.
    
    Regarding `ProjectedTraverser`. Again, a concept internal to Gremlin-Java VM. Note that `Traverser` serialization in GraphSON is not complete regardless (only what is required for practical communication):
    
    ```java
    @Override
            public void serialize(final Traverser traverserInstance, final JsonGenerator jsonGenerator, final SerializerProvider serializerProvider)
                    throws IOException {
                jsonGenerator.writeStartObject();
                jsonGenerator.writeObjectField(GraphSONTokens.BULK, traverserInstance.bulk());
                jsonGenerator.writeObjectField(GraphSONTokens.VALUE, traverserInstance.get());
                jsonGenerator.writeEndObject();
            }
    ```
    
    `ProjectedTraverser` is used by `CollectingBarrierSteps` in OLAP during reductions. For Gremlin-Java VM applications, this is all Gryo serialization stuff with aggregator frameworks.
    
    Finally, regarding testing --- yes, I swapped out `ChainedComparator` for `MultiComparator + ProjectedTraverser` and thus, its tested.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tinkerpop issue #538: TINKERPOP-1248: OrderGlobalStep should use local star ...

Posted by okram <gi...@git.apache.org>.
Github user okram commented on the issue:

    https://github.com/apache/tinkerpop/pull/538
  
    Decided to not touch `GroupStep` till the next release as it is more complicated to handle than `OrderGlobalStep` and `SampleGlobalStep`. Thus, for this PR:
    
    VOTE +1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tinkerpop pull request #538: TINKERPOP-1248: OrderGlobalStep should use loca...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/tinkerpop/pull/538


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] tinkerpop issue #538: TINKERPOP-1248: OrderGlobalStep should use local star ...

Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the issue:

    https://github.com/apache/tinkerpop/pull/538
  
    Code looks good. Minor improvements were discussed in Slack.
    
    VOTE: +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---