[jira] [Commented] (TINKERPOP3-702) Buffer input to inner traversals

["Ran Magen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TINKERPOP3-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694969#comment-14694969 ] 

Ran Magen commented on TINKERPOP3-702:
--------------------------------------

Hey [~okram],  
I don't completely understand why local traversals are needed to reduce memory? Isn't this something that should be handled by the steps inside the traversal?   If the steps "over-reach" by doing something like:  
{code:java}
while(this.starts.hasNext()) { doSomething(this.starts.next()); }
{code}
then there will probably be a memory issue.  
But if the steps act "responsibly" by doing:  
{code:java}
while(this.starts.hasNext() && count < REASONABLE_BULK_SIZE) { doSomething(this.starts.next()); count++; }
{code}
then there shouldn't be a problem. What am I missing here?  

Why should something like {{g.V().repeat(out()).times(4)}} be more memory-limiting then {{g.V().out().out().out().out()}}?  Same goes for {{CoalesceStep}}, {{UnionStep}}, etc.
Bulking a reasonable amount of requests to the BE storage makes a HUGE difference in performance. I believe the current balance between bulking vs. a small memory footprint is skewed too much towards the small memory footprint. Shouldn't vendors have the option to decide whats more important for them?

> Buffer input to inner traversals
> --------------------------------
>
>                 Key: TINKERPOP3-702
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP3-702
>             Project: TinkerPop 3
>          Issue Type: Improvement
>          Components: process
>            Reporter: Ran Magen
>            Assignee: Marko A. Rodriguez
>             Fix For: 3.0.0-incubating
>
>
> In elastic-gremlin we implement an optimized VertexStep. Part of its job is to batch/buffer/bulk different traversers and query them together in-order to minizmize the number of queries. 
> You can see the implementation here: https://github.com/rmagen/elastic-gremlin/blob/master/src/main/java/org/elasticgremlin/process/optimize/ElasticVertexStep.java#L36
> This works great in regular traversals, the "starts" iterator returns as many traversers as the previous step gave out.
> But when the step is in an innerTraversal (e.g. g.V().repeat(__.out()).times(8)), the "starts" iterator only returns one traverser, and will return the next traverser only in the next call to processNextStart. Thus, there is no way to run a bulk query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)