You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by "Marko A. Rodriguez (JIRA)" <ji...@apache.org> on 2015/06/03 00:36:49 UTC
[jira] [Closed] (TINKERPOP3-694) Bulk loading in TP
[ https://issues.apache.org/jira/browse/TINKERPOP3-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marko A. Rodriguez closed TINKERPOP3-694.
-----------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0.GA
Assignee: Marko A. Rodriguez
For this ticket, I did the [~mhfrantz] recommendation for {{AddEdgeStep}} where we now have {{Scope.global}} and {{Scope.local}} semantics. In terms of {{AddVertexStep}} -- I don't think what is being said makes sense. However, please do add a comment if I'm mistaken and {{AddVertexStep}} should be updated. Closing for now.
> Bulk loading in TP
> ------------------
>
> Key: TINKERPOP3-694
> URL: https://issues.apache.org/jira/browse/TINKERPOP3-694
> Project: TinkerPop 3
> Issue Type: Improvement
> Components: process
> Reporter: Ran Magen
> Assignee: Marko A. Rodriguez
> Fix For: 3.0.0.GA
>
>
> Bulk loading can greatly increase performance in many backends, but there is no way for vendors to implement bulk loading in the TP API.
> As for how to do it, I was thinking a vendor could inherit from AddXStep, and implement bulk loading, enabling something like this:
> ArrayList<Object[]> vertices;
> __.inject(vertices).as("x").addV("x")
> Pros:
> 1. This seems like a standard approach to me, since vendors today batch up queries this way (https://github.com/thinkaurelius/titan/blob/titan09/titan-core/src/main/java/com/thinkaurelius/titan/graphdb/tinkerpop/optimize/TitanVertexStep.java#L72)
> 2. In contrast to the Graph.addVertex method (and the likes), using a traversal enables taking advantage of other strategies, like EventStrategy, a validation strategy, etc.
> Cons:
> 1. I don't think the mutating steps currently work with "x" (I'm not sure how you call this feature).
> 2. The steps are marked as final.
> Cheers,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jira] [Closed] (TINKERPOP3-694) Bulk loading in TP
Posted by Matt Frantz <ma...@gmail.com>.
Handling property cardinality is not something that I had thought about.
Perhaps the step arguments could indicate the Cardinality.
...addV(T.label, 'foo', 'a', Cardinality.single, 'b', Cardinality.list,
'c', Cardinality.set, 'd', 'some single constant', 'e', [1,2,3], 'f', new
HashSet([1,2,3]))
For Scope.local, if a Map containing [a:1, b:[4,5,4,7,7], c:[2,3]] were
presented, then the step could do the right thing, i.e. produce a new
vertex with label 'foo', and with properties 'a' through 'f'. Both 'a' and
'd' would be single, 'b' and 'e' would be list, and 'c' and 'f' would be
set.
The Cardinality declarations would indicate that those keys need to be
satisfied by either Map (Scope.local) or path/sideEffect (Scope.global).
On Tue, Jun 2, 2015 at 4:16 PM, Marko Rodriguez <ok...@gmail.com>
wrote:
> Ah. I get it. You guys want the keyValue pairs provided to addV() to be
> variables, not hard coded in the traversal.
>
> It would then be something like this:
>
> g.withSideEffect("m",mapIterator).addV("m")
>
> That is easy enough. Or perhaps, though I'm not sure of the consistency
> (that is, direct complex objects as arguments):
>
> g.addV(mapIterator)
>
> Next, we would need this for AddEdgeStep as well to be consistent:
>
> g.V().as('a').out.addOutE('knows','a', mapIterator)
>
> This is where things get sketchy --- #1, you are not guaranteed an order
> in a traversal so which Map to which Edge? #2 -- we have "addE()" so
> overloaded its crazy. Look at the signature in GraphTraversal. !!! Finally
> #3 -- Map<String,Object> is not rich enough for vertex properties. You need
> a Cardinality, a key, a value, and a pair of key/values. Yes, we have
> Graph.addVertex() which just creates "simple vertex properties", so its
> consistent…but … ?
>
> Anywho, I understand what you are trying to do, however, this
> Iterator<Map<String,Object>> approach seems a bit weird to me…. @dkuppitz?
>
> Thanks,
> Marko.
>
> http://markorodriguez.com
>
> On Jun 2, 2015, at 4:45 PM, Ran Magen <rm...@gmail.com> wrote:
>
> > In addition to what Matt wrote, I think we need a way for a vendor to
> > implement bulk vertex inserts utilizing TINKERPOP3-701.
> > I don't know what's the best way to do that. The best I could come up
> with
> > without changing the API too much is this:
> >
> > {{Iterator<Map> vertices;}}
> > {{__.inject(vertices).as("x").addV("x")}}
> >
> > It's kind of an awkward solution, It'd be good to hear better
> suggestions.
> >
> > In any way, AddEdgeStep looks good!
> > On יום ד׳, 3 ביוני 2015 at 01:37 Marko A. Rodriguez (JIRA) <
> jira@apache.org>
> > wrote:
> >
> >>
> >> [
> >>
> https://issues.apache.org/jira/browse/TINKERPOP3-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> >> ]
> >>
> >> Marko A. Rodriguez closed TINKERPOP3-694.
> >> -----------------------------------------
> >> Resolution: Fixed
> >> Fix Version/s: 3.0.0.GA
> >> Assignee: Marko A. Rodriguez
> >>
> >> For this ticket, I did the [~mhfrantz] recommendation for
> {{AddEdgeStep}}
> >> where we now have {{Scope.global}} and {{Scope.local}} semantics. In
> terms
> >> of {{AddVertexStep}} -- I don't think what is being said makes sense.
> >> However, please do add a comment if I'm mistaken and {{AddVertexStep}}
> >> should be updated. Closing for now.
> >>
> >>> Bulk loading in TP
> >>> ------------------
> >>>
> >>> Key: TINKERPOP3-694
> >>> URL:
> >> https://issues.apache.org/jira/browse/TINKERPOP3-694
> >>> Project: TinkerPop 3
> >>> Issue Type: Improvement
> >>> Components: process
> >>> Reporter: Ran Magen
> >>> Assignee: Marko A. Rodriguez
> >>> Fix For: 3.0.0.GA
> >>>
> >>>
> >>> Bulk loading can greatly increase performance in many backends, but
> >> there is no way for vendors to implement bulk loading in the TP API.
> >>> As for how to do it, I was thinking a vendor could inherit from
> >> AddXStep, and implement bulk loading, enabling something like this:
> >>> ArrayList<Object[]> vertices;
> >>> __.inject(vertices).as("x").addV("x")
> >>> Pros:
> >>> 1. This seems like a standard approach to me, since vendors today batch
> >> up queries this way (
> >>
> https://github.com/thinkaurelius/titan/blob/titan09/titan-core/src/main/java/com/thinkaurelius/titan/graphdb/tinkerpop/optimize/TitanVertexStep.java#L72
> >> )
> >>> 2. In contrast to the Graph.addVertex method (and the likes), using a
> >> traversal enables taking advantage of other strategies, like
> EventStrategy,
> >> a validation strategy, etc.
> >>> Cons:
> >>> 1. I don't think the mutating steps currently work with "x" (I'm not
> >> sure how you call this feature).
> >>> 2. The steps are marked as final.
> >>> Cheers,
> >>
> >>
> >>
> >> --
> >> This message was sent by Atlassian JIRA
> >> (v6.3.4#6332)
> >>
>
>
Re: [jira] [Closed] (TINKERPOP3-694) Bulk loading in TP
Posted by Marko Rodriguez <ok...@gmail.com>.
Ah. I get it. You guys want the keyValue pairs provided to addV() to be variables, not hard coded in the traversal.
It would then be something like this:
g.withSideEffect("m",mapIterator).addV("m")
That is easy enough. Or perhaps, though I'm not sure of the consistency (that is, direct complex objects as arguments):
g.addV(mapIterator)
Next, we would need this for AddEdgeStep as well to be consistent:
g.V().as('a').out.addOutE('knows','a', mapIterator)
This is where things get sketchy --- #1, you are not guaranteed an order in a traversal so which Map to which Edge? #2 -- we have "addE()" so overloaded its crazy. Look at the signature in GraphTraversal. !!! Finally #3 -- Map<String,Object> is not rich enough for vertex properties. You need a Cardinality, a key, a value, and a pair of key/values. Yes, we have Graph.addVertex() which just creates "simple vertex properties", so its consistent…but … ?
Anywho, I understand what you are trying to do, however, this Iterator<Map<String,Object>> approach seems a bit weird to me…. @dkuppitz?
Thanks,
Marko.
http://markorodriguez.com
On Jun 2, 2015, at 4:45 PM, Ran Magen <rm...@gmail.com> wrote:
> In addition to what Matt wrote, I think we need a way for a vendor to
> implement bulk vertex inserts utilizing TINKERPOP3-701.
> I don't know what's the best way to do that. The best I could come up with
> without changing the API too much is this:
>
> {{Iterator<Map> vertices;}}
> {{__.inject(vertices).as("x").addV("x")}}
>
> It's kind of an awkward solution, It'd be good to hear better suggestions.
>
> In any way, AddEdgeStep looks good!
> On יום ד׳, 3 ביוני 2015 at 01:37 Marko A. Rodriguez (JIRA) <ji...@apache.org>
> wrote:
>
>>
>> [
>> https://issues.apache.org/jira/browse/TINKERPOP3-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>> ]
>>
>> Marko A. Rodriguez closed TINKERPOP3-694.
>> -----------------------------------------
>> Resolution: Fixed
>> Fix Version/s: 3.0.0.GA
>> Assignee: Marko A. Rodriguez
>>
>> For this ticket, I did the [~mhfrantz] recommendation for {{AddEdgeStep}}
>> where we now have {{Scope.global}} and {{Scope.local}} semantics. In terms
>> of {{AddVertexStep}} -- I don't think what is being said makes sense.
>> However, please do add a comment if I'm mistaken and {{AddVertexStep}}
>> should be updated. Closing for now.
>>
>>> Bulk loading in TP
>>> ------------------
>>>
>>> Key: TINKERPOP3-694
>>> URL:
>> https://issues.apache.org/jira/browse/TINKERPOP3-694
>>> Project: TinkerPop 3
>>> Issue Type: Improvement
>>> Components: process
>>> Reporter: Ran Magen
>>> Assignee: Marko A. Rodriguez
>>> Fix For: 3.0.0.GA
>>>
>>>
>>> Bulk loading can greatly increase performance in many backends, but
>> there is no way for vendors to implement bulk loading in the TP API.
>>> As for how to do it, I was thinking a vendor could inherit from
>> AddXStep, and implement bulk loading, enabling something like this:
>>> ArrayList<Object[]> vertices;
>>> __.inject(vertices).as("x").addV("x")
>>> Pros:
>>> 1. This seems like a standard approach to me, since vendors today batch
>> up queries this way (
>> https://github.com/thinkaurelius/titan/blob/titan09/titan-core/src/main/java/com/thinkaurelius/titan/graphdb/tinkerpop/optimize/TitanVertexStep.java#L72
>> )
>>> 2. In contrast to the Graph.addVertex method (and the likes), using a
>> traversal enables taking advantage of other strategies, like EventStrategy,
>> a validation strategy, etc.
>>> Cons:
>>> 1. I don't think the mutating steps currently work with "x" (I'm not
>> sure how you call this feature).
>>> 2. The steps are marked as final.
>>> Cheers,
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.4#6332)
>>
Re: [jira] [Closed] (TINKERPOP3-694) Bulk loading in TP
Posted by Ran Magen <rm...@gmail.com>.
In addition to what Matt wrote, I think we need a way for a vendor to
implement bulk vertex inserts utilizing TINKERPOP3-701.
I don't know what's the best way to do that. The best I could come up with
without changing the API too much is this:
{{Iterator<Map> vertices;}}
{{__.inject(vertices).as("x").addV("x")}}
It's kind of an awkward solution, It'd be good to hear better suggestions.
In any way, AddEdgeStep looks good!
On יום ד׳, 3 ביוני 2015 at 01:37 Marko A. Rodriguez (JIRA) <ji...@apache.org>
wrote:
>
> [
> https://issues.apache.org/jira/browse/TINKERPOP3-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Marko A. Rodriguez closed TINKERPOP3-694.
> -----------------------------------------
> Resolution: Fixed
> Fix Version/s: 3.0.0.GA
> Assignee: Marko A. Rodriguez
>
> For this ticket, I did the [~mhfrantz] recommendation for {{AddEdgeStep}}
> where we now have {{Scope.global}} and {{Scope.local}} semantics. In terms
> of {{AddVertexStep}} -- I don't think what is being said makes sense.
> However, please do add a comment if I'm mistaken and {{AddVertexStep}}
> should be updated. Closing for now.
>
> > Bulk loading in TP
> > ------------------
> >
> > Key: TINKERPOP3-694
> > URL:
> https://issues.apache.org/jira/browse/TINKERPOP3-694
> > Project: TinkerPop 3
> > Issue Type: Improvement
> > Components: process
> > Reporter: Ran Magen
> > Assignee: Marko A. Rodriguez
> > Fix For: 3.0.0.GA
> >
> >
> > Bulk loading can greatly increase performance in many backends, but
> there is no way for vendors to implement bulk loading in the TP API.
> > As for how to do it, I was thinking a vendor could inherit from
> AddXStep, and implement bulk loading, enabling something like this:
> > ArrayList<Object[]> vertices;
> > __.inject(vertices).as("x").addV("x")
> > Pros:
> > 1. This seems like a standard approach to me, since vendors today batch
> up queries this way (
> https://github.com/thinkaurelius/titan/blob/titan09/titan-core/src/main/java/com/thinkaurelius/titan/graphdb/tinkerpop/optimize/TitanVertexStep.java#L72
> )
> > 2. In contrast to the Graph.addVertex method (and the likes), using a
> traversal enables taking advantage of other strategies, like EventStrategy,
> a validation strategy, etc.
> > Cons:
> > 1. I don't think the mutating steps currently work with "x" (I'm not
> sure how you call this feature).
> > 2. The steps are marked as final.
> > Cheers,
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>