[jira] [Commented] (TINKERPOP3-694) Bulk loading in TP

["Matt Frantz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TINKERPOP3-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14552972#comment-14552972 ] 

Matt Frantz commented on TINKERPOP3-694:
----------------------------------------

The first novel component of this proposal is concerned with allowing the vendor to specify alternative implementations of mutating steps.  In particular, it could include batching (buffering) to perform multiple mutations in a single backend API call.  (The second novel component is addressed below.)

It looks like the {{Add*Step}} classes each contain two components: (1) the actual mutation logic (delegation to {{Graph}}), and (2) a callback registry.  What if the callback API from {{Mutating}} were given a default implementation, e.g. {{AbstractMutating}} which delegates to a concrete {{CallbackRegistry}}, so that it could be reused to build alternatives to the {{Add*Step}} classes?

The vendor could then provide a complementary {{TraversalStrategy}}, e.g. {{BulkLoadStrategy}}, which would replace the {{Add*Step}} steps with the vendor-specific ones.

We probably want to promote extensibility via composition from abstract classes or via delegation to concrete classes, rather than via inheritance of concrete classes and overriding.  The latter is the hardest to maintain for a framework in the long run, and it would appear that TP2 provided some lessons learned on that.

This might require us to formalize certain elements of the API, e.g. the implied barrier in a buffering mutation step.  The vendor step implementations might be able to implement certain interfaces that signal to other strategies how to deal with the new steps.

The second novel portion of this proposal is to source the mutation parameters from the traverser, rather than having them as step parameters.  This is clearly an enhancement over the existing API, and has less to do with bulk loading per se than about a new functional API.  As such, it can be specified in a vendor-agnostic way.  As proposed, this would be via {{select}} logic.  The proposed syntax might be sugar for "take the {{Map}} in the traverser and turn it into mutation parameters."  Perhaps an example of this canonical form is this:

{noformat}
...select('a','b','c').addV(local)
{noformat}

There could be a combination of "from traverser" and "from step parameters":

{noformat}
...select('a', 'b').addV(local, 'c', 123)
{noformat}

I see no significant benefit to embedding the selection of parameters within the mutating steps.  Doing so violates the separation of concerns for the API.


> Bulk loading in TP
> ------------------
>
>                 Key: TINKERPOP3-694
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP3-694
>             Project: TinkerPop 3
>          Issue Type: Improvement
>          Components: process
>            Reporter: Ran Magen
>
> Bulk loading can greatly increase performance in many backends, but there is no way for vendors to implement bulk loading in the TP API.
> As for how to do it, I was thinking a vendor could inherit from AddXStep, and implement bulk loading, enabling something like this:
> ArrayList<Object[]> vertices;
> __.inject(vertices).as("x").addV("x") 
> Pros:
> 1. This seems like a standard approach to me, since vendors today batch up queries this way (https://github.com/thinkaurelius/titan/blob/titan09/titan-core/src/main/java/com/thinkaurelius/titan/graphdb/tinkerpop/optimize/TitanVertexStep.java#L72)
> 2.  In contrast to the Graph.addVertex method (and the likes), using a traversal enables taking advantage of other strategies, like EventStrategy, a validation strategy, etc.
> Cons:
> 1. I don't think the mutating steps currently work with "x" (I'm not sure how you call this feature).
> 2. The steps are marked as final.
> Cheers,



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)