Re: [TinkerPop] Re: [TinkerPop3] Traversal Strategies, SideEffects, Oh My...

[Stephen Mallette <sp...@gmail.com>]

You can't compare the wrapper approach in TP2 to TP3.  In TP2, the wrapper
was confined to wrapping Blueprints level classes.  Titan was only aware of
Blueprints and all of the other aspects of the stack (e.g. Gremlin) built
on top of that.  In TP3, the Graph implementations have reach into the
query language and the entire API is just "Gremlin".  It's that lack of
separation that creates that problem but it is just that separation that
makes TP3 so much more powerful allowing individual implementations the
opportunity to fully showcase and express their capabilities.

If you really want to dig into the issue Titan is seeing you should take a
look at the code in the the titan09 branch - as a hint on where to start
note this code here:

https://github.com/thinkaurelius/titan/blob/titan09/titan-core/src/main/java/com/thinkaurelius/titan/graphdb/tinkerpop/optimize/TitanVertexStep.java#L72

where one of Titan's traversal optimizer classes encounters a
StrategyVertex instance and converts it to TitanVertex (but never converts
it back - and shouldn't have to know to do so).



On Fri, Jan 23, 2015 at 10:16 AM, Dave <da...@davidcrouchcustomhomes.com>
wrote:

> Hi Stephen,
>
> Regarding the Titan issue, how did it cope with wrappers in TP2?    This
> part of strategy wrappers in TP3 is almost the same so I'm a little
> confused as to what is tripping it up. We are currently using Titan with
> multiple layers of wrappers in TP2 with no problems - although it is an old
> Titan 3.2.  Also, M7 uses sequences of strategies versus the newer nested
> strategies. I'd be curious if the problem still occurs with nested
> strategies.
>
> Thanks,
> Dave
>
>
> On Friday, January 23, 2015 at 4:27:02 AM UTC-8, Stephen Mallette wrote:
>>
>> Marko, do you have any thoughts on Dave's question about extending
>> GraphTraversal?
>>
>> As for the problems Titan ran into with GraphStrategy....it does a lot of
>> optimization on traversals.  In doing so, it encounters the wrapped up
>> instances of graph elements (e.g. StrategyVertex, etc).  To do what it
>> needs to do it needs the base TitanVertex so it unwraps which:
>>
>> 1. bypasses any strategies that were assigned
>> 2. forces Titan to re-wrap the vertex back up
>>
>> We really need to figure out a way to hide such implementation details
>> from graph implementations.
>>
>> Stephen
>>
>> On Fri, Jan 23, 2015 at 12:21 AM, Dave <da...@davidcrouchcustomhomes.com>
>> wrote:
>>
>>> Hi Stephen,
>>>
>>> Thanks for the update. I'm curious as to the GraphStrategy issue that
>>> was found with Titan?
>>>
>>> With regard to feedback from extending graph traversals, to add just a
>>> few custom steps the "of" step works but when the traversal includes
>>> switching back and forth between standard GraphTraversal steps and custom
>>> steps the traversal length and readability becomes a little cumbersome. The
>>> alternative of extending GraphTraversal (like Neo4jGraphTraversal) to make
>>> the custom steps appear native seems like overkill, especially for just a
>>> couple of steps but I guess stubs and maven-exec simplify the process and
>>> it only has to be done once (if I'm not mistaken). I'm also looking at a
>>> couple of other design alternatives.
>>>
>>> Thanks,
>>> Dave
>>>
>>>
>>>
>>> On Thursday, January 22, 2015 at 3:12:41 AM UTC-8, Stephen Mallette
>>> wrote:
>>>>
>>>> Hi Dave, I'm not sure how to advise you at the moment as we found some
>>>> issues with the GraphStrategy design when Titan was being implemented
>>>> against M7.  In Titan's case (and this will apply to any vendor with a
>>>> sufficiently advanced implementation), the current approach forces Titan to
>>>> know something about the various wrapper classes (e.g. StrategyVertex,
>>>> StrategyEdge, etc.).  That was never intended and tells us that something
>>>> is "wrong". As of this moment, I'm not sure what the resolution will be,
>>>> but i must say that it was quite disconcerting to see this problem rise at
>>>> the close of M7.  Obviously, I will post back here as soon as we come to
>>>> some conclusion on what needs to change in order to fix this.
>>>>
>>>> On Tue, Jan 20, 2015 at 11:17 AM, Dave <da...@davidcrouchcustomhomes.
>>>> com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I've read a lot of TP3 code in the past week and now have a good
>>>>> understanding of how DSLs are supported (I answered my questions below). As
>>>>> mentioned below I want to wrap and enhance the underlying graph to support
>>>>> meta-properties on Edges (and possibly hyper-edges). I thought an "abstract
>>>>> edge" could consist of an incoming edge from the source vertex, a vertex
>>>>> holding the properties for the edge, and an outgoing edge to the target
>>>>> vertex.
>>>>>
>>>>> I was going to use a graph strategy to hide its implementation and
>>>>> make it look like a 1st class citizen. I thought I would create an
>>>>> EdgeVertex interface by extending Edge and Vertex, which was appealing
>>>>> because an instance of this interface could be used in place of both an
>>>>> Edge or a Vertex. It seemed like a simple thing to do but I ran into some
>>>>> an obstacle
>>>>>
>>>>> Edge extends EdgeTraversal, which includes methods such as valueMap,
>>>>> PropertyMap and Properties. Vertex extends VertexTraversal and includes the
>>>>> same methods but with different return types. Therefore inheriting from
>>>>> Edge and Vertex causes the "diamond" problem that arises from multiple
>>>>> inheritance. And because the XXXTraversal interface extensions are
>>>>> encapsulated in the Edge and Vertex interfaces I cannot correct the
>>>>> problem.
>>>>>
>>>>> I see two options but would like some feedback:-
>>>>>
>>>>> 1) implement a new "EdgeVertex" concept that is totally custom and
>>>>> somewhat mutually exclusive from Edge and Vertex e.g. g.EV(). v.outEV()
>>>>> etc. But then I cannot use this concept interchangeably with Edge and
>>>>> Vertex and I would be coding a lot of boilerplate.
>>>>>
>>>>> 2) Execute a pull request and modify the tinkerpop code. One proposal
>>>>> is to modify VertexPropertyTraversal so that it extends PropertyTraversal
>>>>> rather than be mutually exclusive of it (this would also make it more
>>>>> consistent - VertexProperty extends Property). This could be achieved by
>>>>> either renaming the clashing methods (valueMap, PropertyMap etc.) or come
>>>>> up with a fancy way to do it with generics e,g, PropertyTraversal<Property>
>>>>> and PropertyTraversal<VertexProperty>. I would also modify the Edge
>>>>> declaration to Edge<T extends Property> so that custom code can pass in an
>>>>> enhanced property interface (e.g. VertexProperty) but the default would be
>>>>> Property, which preserves the Tinkerpop data model.
>>>>>
>>>>> I would like some feedback before I dive into one of the options. Now,
>>>>> I'm leaning towards #2 and also thought about it being my own custom
>>>>> tinkerpop fork but I'm using Titan and was concerned that there could be
>>>>> clashes with Titan's custom Tinkerpop package.
>>>>>
>>>>> Thanks,
>>>>> Dave
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Monday, January 12, 2015 at 12:23:32 PM UTC-8, Dave wrote:
>>>>>>
>>>>>> HI Marko,
>>>>>>
>>>>>> I'm looking at supporting meta-properties on edges (and to possibly
>>>>>> support hyper-edges) and had some questions on TP3 extensibility. For the
>>>>>> "structure" side I will use a graph strategy to change the data model i.e.
>>>>>> an edge property will be equivalent to a vertex property. If successful
>>>>>> then it could be conceivable to go further and make an edge appear like the
>>>>>> vertex interface so I can eventually do things like this for hyper-edges:-
>>>>>>  g.V().outE("has_project").out("permission").has("name", "read").
>>>>>>
>>>>>> I understand the obstacles in the graph strategy functionality to
>>>>>> implement a different data model but think I can get around them. My
>>>>>> questions are more on the "process" side. I would like to extend some of
>>>>>> the standard steps to make the new data model appear native e.g.
>>>>>> outE().out(). Now, I understand that I can write a new DSL but I'm not sure
>>>>>> this is exactly what I want (unless I can inherit and extend the standard
>>>>>> one). You mention quite a bit about traversal strategies below but its not
>>>>>> clear if these are what I need either (maybe for some things).
>>>>>>
>>>>>> 1) I assume only one DSL can be "active" at one time i.e. the steps
>>>>>> in graph.of(MyDSL.class) are mutually exclusive of the standard graph
>>>>>> traversal steps - I cannot do things like g.V().out().myStep().out() ?
>>>>>> 2) How do I extend the standard DSL?   I assume I create a new
>>>>>> interface that inherits from GraphTraversal but what do I need to do for
>>>>>> the implementation besides the return type for my steps being
>>>>>> "MyGraphTraversal"?
>>>>>> 3) Is there a GraphTraversal implementation class I can extend to add
>>>>>> new steps and keep the standard steps?
>>>>>> 4) If yes then I assume I can I override the existing steps to change
>>>>>> their behavior?
>>>>>>
>>>>>> Lastly, I have not yet used any of the graph computer/OLAP
>>>>>> functionality. If I can extend the "structure" to change the underlying
>>>>>> data model, and assuming I can change/enhance the graph traversal steps on
>>>>>> the "process" side to support my data model, will this break the OLAP/graph
>>>>>> computer functionality (I assume the out-of-the-box programs could break) ?
>>>>>>
>>>>>> Thanks,
>>>>>> Dave
>>>>>>
>>>>>>
>>>>>> On Thursday, January 8, 2015 at 2:09:50 PM UTC-8, Marko A. Rodriguez
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I was originally going to write this to a few people, but there are
>>>>>>> probably others out there that might find this useful (e.g. SQLg,
>>>>>>> Gremlin-Scala, etc.). This is pretty low-level TP3, so most people can just
>>>>>>> ignore this.
>>>>>>>
>>>>>>> -------
>>>>>>>
>>>>>>> TraversalStrategies are now cached according to the concept of the
>>>>>>> "emanating class," not the traversal class. The benefit of this is two fold:
>>>>>>>
>>>>>>> 1. Unless the vendor is extending GraphTraversal with more methods,
>>>>>>> vendors no longer need to create a DefaultVendorGraphTraversal as they can
>>>>>>> rely on TinkerPop's DefaultGraphTraversal.
>>>>>>> 2. The vendor implementations can rely on the interface default
>>>>>>> implementations -- e.g. g.V(), g.E(), vertex.start(), edge.start(),
>>>>>>> vertexProperty.start().
>>>>>>>
>>>>>>> https://github.com/tinkerpop/tinkerpop3/blob/master/gremlin-
>>>>>>> core/src/main/java/com/tinkerpop/gremlin/process/graph/util/
>>>>>>> DefaultGraphTraversal.java
>>>>>>> https://github.com/tinkerpop/tinkerpop3/blob/master/gremlin-
>>>>>>> core/src/main/java/com/tinkerpop/gremlin/process/util/
>>>>>>> DefaultTraversal.java#L29-L31
>>>>>>>
>>>>>>> If a vendor has a strategy they want to register, they will do it as
>>>>>>> follows:
>>>>>>>
>>>>>>> https://github.com/tinkerpop/tinkerpop3/blob/master/tinkergr
>>>>>>> aph-gremlin/src/main/java/com/tinkerpop/gremlin/tinkergraph/
>>>>>>> structure/TinkerGraph.java#L44-L52
>>>>>>>
>>>>>>> There is no more GraphTraversalStrategyRegistry. If a user really
>>>>>>> wants to update the strategies of a particular "emanating class," they
>>>>>>> simply interact with the cache directly.
>>>>>>>
>>>>>>> https://github.com/tinkerpop/tinkerpop3/blob/master/gremlin-
>>>>>>> core/src/main/java/com/tinkerpop/gremlin/process/TraversalSt
>>>>>>> rategies.java#L121
>>>>>>>
>>>>>>> Most users will never do this. Vendors will only do this to get some
>>>>>>> base strategies that they might want before adding their own and pushing it
>>>>>>> back down to the cache. Next, notice how AnonymousGraphTraversal does not
>>>>>>> have any strategies defined for it.
>>>>>>>
>>>>>>> https://github.com/tinkerpop/tinkerpop3/blob/master/gremlin-
>>>>>>> core/src/main/java/com/tinkerpop/gremlin/process/TraversalSt
>>>>>>> rategies.java#L144
>>>>>>>
>>>>>>> Finally, if a vendor doesn't have any strategies and they just want
>>>>>>> to rely on the core TinkerPop strategies, they need not do anything special.
>>>>>>>
>>>>>>> https://github.com/tinkerpop/tinkerpop3/blob/master/gremlin-
>>>>>>> core/src/main/java/com/tinkerpop/gremlin/process/TraversalSt
>>>>>>> rategies.java#L144
>>>>>>>
>>>>>>> Next, there is no need for a vendor to create a
>>>>>>> DefaultVendorGraphTraversal instance. The only vendor that needs it is
>>>>>>> Neo4j because of the extended cypher()-step (Neo4j has
>>>>>>> DefaultNeo4jGraphTraversal for this reason). If the vendor only uses the
>>>>>>> core steps in GraphTraversal, then they need not even reference the
>>>>>>> construction of a traversal at all -- let the defaults of the respective
>>>>>>> interfaces handle it.
>>>>>>>
>>>>>>> https://github.com/tinkerpop/tinkerpop3/blob/master/gremlin-
>>>>>>> core/src/main/java/com/tinkerpop/gremlin/process/graph/
>>>>>>> VertexTraversal.java#L17-L20
>>>>>>> https://github.com/tinkerpop/tinkerpop3/blob/master/gremlin-
>>>>>>> core/src/main/java/com/tinkerpop/gremlin/process/graph/
>>>>>>> EdgeTraversal.java#L15-L19
>>>>>>> https://github.com/tinkerpop/tinkerpop3/blob/master/gremlin-
>>>>>>> core/src/main/java/com/tinkerpop/gremlin/process/graph/
>>>>>>> VertexPropertyTraversal.java#L15-L19
>>>>>>>
>>>>>>> …where Vertex extends VertexTraversal (thus, these are VendorVertex
>>>>>>> methods).
>>>>>>>
>>>>>>> More stuff. There is no loner Graph.of(). With the inclusion of
>>>>>>> AnonymousGraphTraversal.Tokens.__, Graph.of() went away. However,
>>>>>>> Graph.of(Traversal.class) still exists and is still the means by which
>>>>>>> developer DSLs are created -- e.g. g.of(SocialTraversal.class).pe
>>>>>>> rson('marko').who().knows().person('bob').
>>>>>>>
>>>>>>> And finally, I think, with DefaultGraphTraversal being so simple and
>>>>>>> only requiring knowledge of the class that spawns it (the "emanating
>>>>>>> class"), there is no need to pass around a Graph reference. This Graph
>>>>>>> reference was used to populate Traversal.getSideEffects().setGraph().
>>>>>>> getGraph() was only used in a few places with workaround easily
>>>>>>> accomplished. Having the Graph in the sideEffect is dangerous for a few
>>>>>>> reasons -- 1.) not all traversals spawn from graph objects so we shouldn't
>>>>>>> have step developers think its always there, 2.) when you go to OLAP the
>>>>>>> graph reference is meaningless and thus, must be removed prior to
>>>>>>> submission to the cluster (error prone), 3.) with it gone, methods go away
>>>>>>> and there is less to go wrong.
>>>>>>>
>>>>>>> I hope that helps people out. There is still some more work I need
>>>>>>> to do on TraverserGenerators (but I don't think there are any vendors even
>>>>>>> looking into that are simply using the core SimpleTraverser and
>>>>>>> PathTraverser).
>>>>>>>
>>>>>>> Take care,
>>>>>>> Marko.
>>>>>>>
>>>>>>> http://markorodriguez.com
>>>>>>>
>>>>>>>  --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Gremlin-users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to gremlin-user...@googlegroups.com.
>>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>>> msgid/gremlin-users/ae67c5a0-18bd-41dc-8929-6b249a3ba19a%40goo
>>>>> glegroups.com
>>>>> <https://groups.google.com/d/msgid/gremlin-users/ae67c5a0-18bd-41dc-8929-6b249a3ba19a%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "Gremlin-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to gremlin-user...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/gremlin-users/28d2603c-1865-42f8-a58a-3c4ac31837c7%
>>> 40googlegroups.com
>>> <https://groups.google.com/d/msgid/gremlin-users/28d2603c-1865-42f8-a58a-3c4ac31837c7%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gremlin-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gremlin-users/d53246f8-af3e-4982-814b-57d6bc2d4c4d%40googlegroups.com
> <https://groups.google.com/d/msgid/gremlin-users/d53246f8-af3e-4982-814b-57d6bc2d4c4d%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>