[jira] [Commented] (TINKERPOP3-319) BulkLoaderVertexProgram for generalized batch loading across graphs

["Matthias Broecheler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TINKERPOP3-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876857#comment-14876857 ] 

Matthias Broecheler commented on TINKERPOP3-319:
------------------------------------------------

The following optimizations should be implemented to improve the performance of BLVP:

* In line 212, BLVP should get the information whether the vertex was created or retrieved. If it was created (i.e. it did not exist before) then we are guaranteed that it cannot have any vertex properties. As such, the BLVP should then just create the vertex properties without checking for their existence first - this will be significantly faster.
* Similarly, when loading edges in the second iteration, it should first compute this boolean variable {{requiresIncremental = sourceVertex.edges(OUT).hasNext() && outV.edges(OUT).hasNext()}} and then only do incremental loading on edges if this variable is true. If it is not true incremental loading (i.e. checking for edge existence) isn't necessary.

Both improvement together should dramatically improve the performance of BLVP since it will require a read per edge/vertex property only in those cases where a previous job failed. Under "normal" operational conditions it only requires one read per vertex per iteration. That is, the reads scale in O(|V|) and not O(|E|).

In addition, there should be an option for IncrementalBulkLoader so that it does not attempt to update edges and vertex properties when those already exist. In most cases, the edge will be identical when it has been loaded in a previous job (since edge and property mutations are atomic in most graph databases) and hence this check is unnecessary and being able to make it optional can save time.

> BulkLoaderVertexProgram for generalized batch loading across graphs
> -------------------------------------------------------------------
>
>                 Key: TINKERPOP3-319
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP3-319
>             Project: TinkerPop 3
>          Issue Type: Improvement
>          Components: process
>    Affects Versions: 3.0.1-incubating
>            Reporter: Marko A. Rodriguez
>            Assignee: Daniel Kuppitz
>             Fix For: 3.1.0-incubating, 3.0.2-incubating
>
>
> After working on {{BulkLoaderVertexProgram}} for Titan, it is trivial to add this generally to TinkerPop -- equivalent to BlueprintsOutputFormat (or whatever the bulk loader was known that was blueprints specific). However, given that Titan and TinkerPop have the same data model, Titan having its own {{BulkLoaderVertexProgram}} isn't necessary as there is no longer a data model alignment issue. The difference would be that instead of:
> {code:groovy}
> g.V.compute().program(BulkLoaderVertexProgram.build().titan(propertiesFile).create()).submit()
> {code}
> It would simply be:
> {code:groovy}
> g.V.compute().program(BulkLoaderVertexProgram.build().factory(propertiesFile).create()).submit()
> {code}
> ...and {{BulkLoaderVertexProgram}} would use {{GraphFactory.open()}} to instantiate the connection to the graph. Moreover, (and [~spmallette] will need to clear my head here), if the factory opened up a Gremlin Server connection, then we get parallel writing to embedded graph databases like Neo4j.
> {{BulkLoaderVertexProgram}} is simply a vertex program that parallel loads a graph (with a graph computer) to any other graph that can be accessed via {{GraphFactory}} (which is every TP3 graph).
> [~dalaro] @mbroecheler [~dkuppitz] 
> EXTENDED NOTES:
> * {{SchemaInference}} would be a MapReduce job executed prior to {{BulkLoaderVertexProgram}}
> * Titan and Neo4j can each have their own {{SchemaInference}} implementations.
> * Incremental loading .... I forget how this worked.
> * Bulk mutations ... this is possible at the TP3 level with hidden properties and smart add/remove/etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)