[DISCUSS] TinkerPop mantained Gremlin Language Variants

[Stephen Mallette <sp...@gmail.com>]

This thread on Gremlin Language Variants has been very interesting:

https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc

I think that this work goes a long way to address two issues I've been
concerned about:

1. Greater consistency in how different languages do Gremlin
2. Less fragmentation in terms of libraries and how they work so that users
aren't confused with how to get started (though I don't think the goal here
is to restrict choices or slow down innovation)

One of the first things we should probably do is start thinking in terms of
the types of libraries that are built on TinkerPop (outside of those things
that are Graph Systems) and those are listed here currently:

http://tinkerpop.apache.org/#graph-libraries

Marko mentioned to me that he saw the libraries we listed here breaking
into three categories:

1. Gremlin Language Variants - which the other thread demonstrates quite
nicely
2. Gremlin Drivers - the Gremlin Server protocol implementations - those
things that send traversals to Gremlin Server and get back results.
3. OGM and others - I say "others" because there might be plugins and other
similar odds and ends

I like Marko's category system here and I think that having these kinds of
categories will help folks organize their libraries to fit into one of
these spaces and make it easier for users to know what they need to get in
order to start doing TinkerPop in their language.

Anyway, the category thing is just setting the stage for this big
bombshell.  I think TinkerPop should consider maintaining the Gremlin
Language Variants.

Heresy! right?

Well, I think it's the best way to achieve consistency across languages.
Under this model, TinkerPop provides the base language variant and people
can choose to extend upon it, but the base stays tied to our archetype of
Java and we end up with a much more clear story for virtually any
programming language.

So how do we do this? Slowly and deliberately. We should look to only
include language variants where we:

+ have good automation in place (like what Marko did for Python),
+ some competence on the committer list in that language
+ a nice testing framework that operates in our standard build/release
process.

That's setting a high bar, but if we don't keep it high, I think we will be
left unable to properly support and maintain what we hang out there.

I'd also like to express that we should not be looking to maintain language
drivers. I think that should remain a third-party community effort just
like Graph Systems. In other words, we remain a repository for reference
implementations for everything else. Why? Because, as it sits right now,
just based on the level of effort for what Marko did with Python,
maintaining a "base" Gremlin Language Variants shouldn't be hard. We won't
be building tons of add-on capabilities to the base variants - they will
pretty much just stay on par with the java archetype.  Drivers on the other
hand have lots of implementation details, with many different technologies
that could be used, etc.  They have similar complexity to Graph System
implementations in many ways. I also think that the drivers can afford to
have different APIs and approaches without being detrimental to the
community.

If gremlin-js-driver wants to do:

client.submit("g.V()")

and gremlin-python-driver wants to do:

client.send("g.V()")

that's not a big deal.

The last point that I'll make is that I think Gremlin Language Variants,
that don't operate on the JVM (e.g. Jython) and use Gremlin Server, should
have some abstraction that is similar to RemoteGraph.  RemoteGraph exposes
a DriverConnection interface that is currently implemented by
gremlin-driver.  The DriverConnection is responsible for sending a
traversal to the server and returning results. It would be nice if the
language variants had a similar interface that the various community
drivers could implement. In that way, the user never has to do any form of:

client.submit(someGremlinString)

in any language. We really need to try to make that pattern go away across
the TinkerPop community.

So - that's was a long email. Looking forward to hearing some discussion on
this.

Stephen

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Stephen Mallette <sp...@gmail.com>]

The use of an index for injection was just for purpose of demonstrating
things quickly. You could maybe use an identity step or something as a
placeholder and then inject/replace on that step instance.  It's not
perfect - just thinking out loud. There is a ticket out there to make
TraversalStrategy writing "better". Ultimately, anything we do in that
regard would probably help this problem.

On Thu, Apr 21, 2016 at 10:32 AM, Dylan Millikin <dy...@gmail.com>
wrote:

> Ohh, I wasn't aware of that. It looks like TraversalHelper has a bunch of
> really useful stuff for these scenarios.
> You are indeed grasping the problem.
> Looking at the signatures there are still grey areas and things that aren't
> really usable (like having to know the index of the injection point) and
> I'm not really sure how this translates to sub traversals like:
>
> g.V().hasLabel("project").union(out('dev')/** inject here **/,
> out('company'))
> yet still work with:
> g.V().hasLabel("person")/** inject here **/.out('friend').out('company');
>
> Is there perhaps a way of including a flag somewhere and have that server
> as the index id ?
>
> Anyways this is definitely what I'm talking about. I'm guessing that there
> may be some conflicts from the fact that method signatures aren't
> necessarily kept in gremlin language variants. Mostly around changing
> existing steps, but if there's a way to get this to work in broader use
> cases it would definitely fit the bill.
>
> Cheers for that. I'll definitely dig a bit deeper around this.
>
> On Thu, Apr 21, 2016 at 10:18 AM, Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > I'm not sure what you mean by:
> >
> > > Why is this not an option: It boils down to the fact that you still
> > can't modify
> > a traversal "after the fact".
> >
> > You can modify the Traversal up to the point where you iterate it and
> > TraversalStrategy are applied.
> >
> > gremlin> base = g.V().hasLabel('person').out();[]
> > gremlin> filter = __.has('age',29)
> > gremlin> TraversalHelper.insertTraversal(1,filter,base);[]
> > gremlin> base.toString()
> > ==>[GraphStep(vertex,[]), HasStep([~label.eq(person)]),
> > HasStep([age.eq(29)]), VertexStep(OUT,vertex)]
> > gremlin> base
> > ==>v[3]
> > ==>v[2]
> > ==>v[4]
> >
> > Note how I injected an anonymous Traversal (i.e. the "filter") into the
> > middle of a "base" Traversal. This is a little known function that is
> > probably familiar to TraversalStrategy developers only, but it exists. I
> > suppose more advanced forms of Gremlin Language Variants should/could
> have
> > this kind of capability - I guess the point is that it's possible and
> > arguably better than string manipulation. You might not agree, but am I
> at
> > least grasping the problem you're trying to express?
> >
> >
> > On Thu, Apr 21, 2016 at 9:49 AM, Dylan Millikin <
> dylan.millikin@gmail.com>
> > wrote:
> >
> > > Essentially what is required is to be able to "build" traversals with a
> > > complete disregard for the order of the steps you're adding. Think of
> it
> > as
> > > adding entries to a match(). It doesn't matter what order you add those
> > > entries in.
> > > This is something that is very "declarative" in nature but is a
> > requirement
> > > when developing complex applications that need modularity. You also
> need
> > > this in an imperative setting as you may want control on how the data
> is
> > > fetched.
> > >
> > > On Thu, Apr 21, 2016 at 9:39 AM, Dylan Millikin <
> > dylan.millikin@gmail.com>
> > > wrote:
> > >
> > > > Yeah there are some more complex situations. You actually left out
> the
> > > > final out() in your example. For instance just keeping things simple,
> > > > your base query could be required to be reused in many places. So the
> > > > following is not an option:
> > > >
> > > > base = g.V().hasLabel('person');[]
> > > > /** apply some filters like you did **/
> > > > base.out('company')
> > > >
> > > > Why is this not an option: It boils down to the fact that you still
> > can't
> > > > modify a traversal "after the fact".
> > > > Your reusable base is not complete without the out() step. Therefore,
> > > > using this model you can't store the base query and test it
> separately.
> > > You
> > > > also have to append the out step everywhere you use this base, which
> in
> > > > addition to creating a lot of duplicate code can also not be an
> option
> > > when
> > > > queries are generated automatically. Sometimes you don't even really
> > know
> > > > what the base query is, you just know that it contains certain
> > "injection
> > > > points".
> > > > To give you a few examples, these following base queries all have the
> > > same
> > > > injection point that allows them to be edited with the same filters:
> > > >
> > > > g.V().hasLabel("person")/** inject here **/.out('company');
> > > > g.V().hasLabel("person")/** inject here
> > **/.out('friend').out('company');
> > > > g.V().has("person", "name", "marko").out('friend')/** inject here
> > > > **/.out('company');
> > > >
> > > > These are all valid. My filters apply to "people" but the traversals
> > > > leading to and from these people can be anything. And
> programmatically
> > I
> > > > have no way of knowing what the traversal is. This gets increasingly
> > > > complexe with injection points in sub-traversals, or when injecting
> > full
> > > > traversals with their own injection points, and so on.
> > > >
> > > > Also your example doesn't cover changing existing steps.
> > > >
> > > > I don't know if that sheds anymore light.
> > > >
> > > > On Thu, Apr 21, 2016 at 9:14 AM, Stephen Mallette <
> > spmallette@gmail.com>
> > > > wrote:
> > > >
> > > >> So "query building" is where you hold string representations of
> steps
> > or
> > > >> groups of steps and then have some logic that concatenates them
> > > together.
> > > >> I
> > > >> guess my question is why are "strings" a requirement for that? It
> > seems
> > > >> like you could do the same query builder stuff with an actual
> > Traversal
> > > >> object in whatever Gremlin Language Variant you were working with.
> How
> > > is
> > > >> string concatenation different than this:
> > > >>
> > > >> gremlin> graph = TinkerFactory.createModern()
> > > >> ==>tinkergraph[vertices:6 edges:6]
> > > >> gremlin> g = graph.traversal()
> > > >> ==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
> > > >> gremlin> base = g.V().hasLabel('person');[]
> > > >> gremlin> addFilters = { age, name, t ->
> > > >> gremlin>   t = age > 0 ? t.has('age',age) : t
> > > >> gremlin>   !name.isEmpty() ? t.has('name',name) : t
> > > >> gremlin> }
> > > >> ==>groovysh_evaluate$_run_closure1@503d56b5
> > > >> gremlin> traversal = addFilters(29,'',base);[]
> > > >> gremlin> traversal.toString()
> > > >> ==>[GraphStep(vertex,[]), HasStep([~label.eq(person)]),
> > > >> HasStep([age.eq(29)])]
> > > >> gremlin> traversal
> > > >> ==>v[1]
> > > >>
> > > >> What about DSLs? Maybe filtering logic drops behind a custom step
> > > specific
> > > >> to the domain:
> > > >>
> > > >> g.V().personsWith(29, '')
> > > >>
> > > >> which would basically compile to the same thing as the gremlin
> output
> > > >> above:
> > > >>
> > > >> [GraphStep(vertex,[]), HasStep([~label.eq(person)]),
> > > >> HasStep([age.eq(29)])]
> > > >>
> > > >> Is there some more complex aspect "query building" that can't be
> > handled
> > > >> this way (I know I took a fairly simple example)?
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Apr 20, 2016 at 7:54 PM, Dylan Millikin <
> > > dylan.millikin@gmail.com
> > > >> >
> > > >> wrote:
> > > >>
> > > >> > I've approached this a few times in the past though never really
> in
> > > >> depth.
> > > >> > The idea is that you want separation of logic in your queries,
> > mostly
> > > >> for
> > > >> > maintenance, testing and overall convenience. This takes the form
> of
> > > >> > partial/incomplete traversals that have no start nor end.
> > > >> > One of the simplest applications would be to have a base query and
> > > apply
> > > >> > filters to this query. For example we can imagine a recap page of
> > > >> > everyone's companies:
> > > >> >
> > > >> > base query : g.V().has(label, "person").out('company')
> > > >> >
> > > >> > To this you have a set of filters that are partial traversals and
> > > allow
> > > >> > users to better refine their search:
> > > >> >
> > > >> > older than 30 filter: has("age", gt(30))
> > > >> > male filter: has("gender", "male")
> > > >> >
> > > >> > Now depending on user input you'll want to apply either or both of
> > the
> > > >> > above to "person" in order to obtain something along the lines of:
> > > >> >
> > > >> > g.V().has(label, "person").has("age", gt(30)).has("gender",
> > > >> > "male").out('company')
> > > >> >
> > > >> > Of course there are plenty of ways of doing the above depending on
> > > >> > requirements and complexity. Such as where() or
> > as("a")....select("a")
> > > >> > etc..
> > > >> >
> > > >> > These are simple examples where steps are appended in various
> places
> > > but
> > > >> > you can imagine the same with traversal manipulation such as
> turning
> > > >> > out('company') into out('company', 'organization') where the step
> > gets
> > > >> > altered. This would work the same way if you used
> > > union(out('company'))
> > > >> > instead. You would have to alter the union step like this :
> > > >> > union(out('company'),
> > > >> > out('organization'))
> > > >> >
> > > >> > All of the above are relatively simple but picture having a lot of
> > > very
> > > >> > complex traversal "filters" that sometimes have dependencies on
> > other
> > > >> > filters.
> > > >> > All this is doable but it currently requires looking at the
> gremlin
> > > >> > building process as a query building one rather than native
> support
> > > for
> > > >> the
> > > >> > gremlin language.
> > > >> >
> > > >> > Does that make any sense?
> > > >> >
> > > >> > On Wed, Apr 20, 2016 at 7:15 PM, Stephen Mallette <
> > > spmallette@gmail.com
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> > > >  It's imperative that you have the ability to group multiline
> > > >> scripts
> > > >> > > into a single query or the reactivity of your applications will
> > > >> greatly
> > > >> > > suffer.
> > > >> > >
> > > >> > > A fair point and something we might yet address as part of all
> > this
> > > >> > > thinking. RemoteGraph is pretty new and it demonstrated a
> critical
> > > >> aspect
> > > >> > > of communications with Gremlin Server. Now we need to think
> about
> > > how
> > > >> to
> > > >> > > improve upon it.
> > > >> > >
> > > >> > > > Namely having the ability to edit/modify/add steps to an
> already
> > > >> > defined
> > > >> > > traversal.
> > > >> > >
> > > >> > > I don't quite follow that point...could you please elaborate?
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Apr 20, 2016 at 4:15 PM, Dylan Millikin <
> > > >> > dylan.millikin@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Nice post,
> > > >> > > >
> > > >> > > > I'm going to jump straight to the end. I think that one of the
> > > >> problems
> > > >> > > of
> > > >> > > > doing an abstraction similar to remoteGraph is that in reality
> > the
> > > >> > > overhead
> > > >> > > > of communicating with the server is too big to make this
> viable.
> > > >> It's
> > > >> > > > imperative that you have the ability to group multiline
> scripts
> > > >> into a
> > > >> > > > single query or the reactivity of your applications will
> greatly
> > > >> > suffer.
> > > >> > > >
> > > >> > > > The Gremlin-java base also has a few limitations that some
> query
> > > >> > builders
> > > >> > > > try to fix (which can only be done if you abandon the idea of
> a
> > > >> natural
> > > >> > > > gremlin language variant in favor of a query builder). Namely
> > > having
> > > >> > the
> > > >> > > > ability to edit/modify/add steps to an already defined
> > traversal.
> > > >> > Though
> > > >> > > in
> > > >> > > > time it might be nice to have these be part of the original
> > > >> > > implementation.
> > > >> > > >
> > > >> > > > I personally love the idea of gremlin language variants. I
> just
> > > >> don't
> > > >> > > think
> > > >> > > > they're production value is any good without some extended
> > > >> > functionality
> > > >> > > > (beyond what gremlin-java currently is).
> > > >> > > >
> > > >> > > > On Wed, Apr 20, 2016 at 3:23 PM, Stephen Mallette <
> > > >> > spmallette@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > This thread on Gremlin Language Variants has been very
> > > >> interesting:
> > > >> > > > >
> > > >> > > > > https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
> > > >> > > > >
> > > >> > > > > I think that this work goes a long way to address two issues
> > > I've
> > > >> > been
> > > >> > > > > concerned about:
> > > >> > > > >
> > > >> > > > > 1. Greater consistency in how different languages do Gremlin
> > > >> > > > > 2. Less fragmentation in terms of libraries and how they
> work
> > so
> > > >> that
> > > >> > > > users
> > > >> > > > > aren't confused with how to get started (though I don't
> think
> > > the
> > > >> > goal
> > > >> > > > here
> > > >> > > > > is to restrict choices or slow down innovation)
> > > >> > > > >
> > > >> > > > > One of the first things we should probably do is start
> > thinking
> > > in
> > > >> > > terms
> > > >> > > > of
> > > >> > > > > the types of libraries that are built on TinkerPop (outside
> of
> > > >> those
> > > >> > > > things
> > > >> > > > > that are Graph Systems) and those are listed here currently:
> > > >> > > > >
> > > >> > > > > http://tinkerpop.apache.org/#graph-libraries
> > > >> > > > >
> > > >> > > > > Marko mentioned to me that he saw the libraries we listed
> here
> > > >> > breaking
> > > >> > > > > into three categories:
> > > >> > > > >
> > > >> > > > > 1. Gremlin Language Variants - which the other thread
> > > demonstrates
> > > >> > > quite
> > > >> > > > > nicely
> > > >> > > > > 2. Gremlin Drivers - the Gremlin Server protocol
> > > implementations -
> > > >> > > those
> > > >> > > > > things that send traversals to Gremlin Server and get back
> > > >> results.
> > > >> > > > > 3. OGM and others - I say "others" because there might be
> > > plugins
> > > >> and
> > > >> > > > other
> > > >> > > > > similar odds and ends
> > > >> > > > >
> > > >> > > > > I like Marko's category system here and I think that having
> > > these
> > > >> > kinds
> > > >> > > > of
> > > >> > > > > categories will help folks organize their libraries to fit
> > into
> > > >> one
> > > >> > of
> > > >> > > > > these spaces and make it easier for users to know what they
> > need
> > > >> to
> > > >> > get
> > > >> > > > in
> > > >> > > > > order to start doing TinkerPop in their language.
> > > >> > > > >
> > > >> > > > > Anyway, the category thing is just setting the stage for
> this
> > > big
> > > >> > > > > bombshell.  I think TinkerPop should consider maintaining
> the
> > > >> Gremlin
> > > >> > > > > Language Variants.
> > > >> > > > >
> > > >> > > > > Heresy! right?
> > > >> > > > >
> > > >> > > > > Well, I think it's the best way to achieve consistency
> across
> > > >> > > languages.
> > > >> > > > > Under this model, TinkerPop provides the base language
> variant
> > > and
> > > >> > > people
> > > >> > > > > can choose to extend upon it, but the base stays tied to our
> > > >> > archetype
> > > >> > > of
> > > >> > > > > Java and we end up with a much more clear story for
> virtually
> > > any
> > > >> > > > > programming language.
> > > >> > > > >
> > > >> > > > > So how do we do this? Slowly and deliberately. We should
> look
> > to
> > > >> only
> > > >> > > > > include language variants where we:
> > > >> > > > >
> > > >> > > > > + have good automation in place (like what Marko did for
> > > Python),
> > > >> > > > > + some competence on the committer list in that language
> > > >> > > > > + a nice testing framework that operates in our standard
> > > >> > build/release
> > > >> > > > > process.
> > > >> > > > >
> > > >> > > > > That's setting a high bar, but if we don't keep it high, I
> > think
> > > >> we
> > > >> > > will
> > > >> > > > be
> > > >> > > > > left unable to properly support and maintain what we hang
> out
> > > >> there.
> > > >> > > > >
> > > >> > > > > I'd also like to express that we should not be looking to
> > > maintain
> > > >> > > > language
> > > >> > > > > drivers. I think that should remain a third-party community
> > > effort
> > > >> > just
> > > >> > > > > like Graph Systems. In other words, we remain a repository
> for
> > > >> > > reference
> > > >> > > > > implementations for everything else. Why? Because, as it
> sits
> > > >> right
> > > >> > > now,
> > > >> > > > > just based on the level of effort for what Marko did with
> > > Python,
> > > >> > > > > maintaining a "base" Gremlin Language Variants shouldn't be
> > > hard.
> > > >> We
> > > >> > > > won't
> > > >> > > > > be building tons of add-on capabilities to the base
> variants -
> > > >> they
> > > >> > > will
> > > >> > > > > pretty much just stay on par with the java archetype.
> Drivers
> > > on
> > > >> the
> > > >> > > > other
> > > >> > > > > hand have lots of implementation details, with many
> different
> > > >> > > > technologies
> > > >> > > > > that could be used, etc.  They have similar complexity to
> > Graph
> > > >> > System
> > > >> > > > > implementations in many ways. I also think that the drivers
> > can
> > > >> > afford
> > > >> > > to
> > > >> > > > > have different APIs and approaches without being detrimental
> > to
> > > >> the
> > > >> > > > > community.
> > > >> > > > >
> > > >> > > > > If gremlin-js-driver wants to do:
> > > >> > > > >
> > > >> > > > > client.submit("g.V()")
> > > >> > > > >
> > > >> > > > > and gremlin-python-driver wants to do:
> > > >> > > > >
> > > >> > > > > client.send("g.V()")
> > > >> > > > >
> > > >> > > > > that's not a big deal.
> > > >> > > > >
> > > >> > > > > The last point that I'll make is that I think Gremlin
> Language
> > > >> > > Variants,
> > > >> > > > > that don't operate on the JVM (e.g. Jython) and use Gremlin
> > > >> Server,
> > > >> > > > should
> > > >> > > > > have some abstraction that is similar to RemoteGraph.
> > > RemoteGraph
> > > >> > > > exposes
> > > >> > > > > a DriverConnection interface that is currently implemented
> by
> > > >> > > > > gremlin-driver.  The DriverConnection is responsible for
> > > sending a
> > > >> > > > > traversal to the server and returning results. It would be
> > nice
> > > if
> > > >> > the
> > > >> > > > > language variants had a similar interface that the various
> > > >> community
> > > >> > > > > drivers could implement. In that way, the user never has to
> do
> > > any
> > > >> > form
> > > >> > > > of:
> > > >> > > > >
> > > >> > > > > client.submit(someGremlinString)
> > > >> > > > >
> > > >> > > > > in any language. We really need to try to make that pattern
> go
> > > >> away
> > > >> > > > across
> > > >> > > > > the TinkerPop community.
> > > >> > > > >
> > > >> > > > > So - that's was a long email. Looking forward to hearing
> some
> > > >> > > discussion
> > > >> > > > on
> > > >> > > > > this.
> > > >> > > > >
> > > >> > > > > Stephen
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Dylan Millikin <dy...@gmail.com>]

Ohh, I wasn't aware of that. It looks like TraversalHelper has a bunch of
really useful stuff for these scenarios.
You are indeed grasping the problem.
Looking at the signatures there are still grey areas and things that aren't
really usable (like having to know the index of the injection point) and
I'm not really sure how this translates to sub traversals like:

g.V().hasLabel("project").union(out('dev')/** inject here **/,
out('company'))
yet still work with:
g.V().hasLabel("person")/** inject here **/.out('friend').out('company');

Is there perhaps a way of including a flag somewhere and have that server
as the index id ?

Anyways this is definitely what I'm talking about. I'm guessing that there
may be some conflicts from the fact that method signatures aren't
necessarily kept in gremlin language variants. Mostly around changing
existing steps, but if there's a way to get this to work in broader use
cases it would definitely fit the bill.

Cheers for that. I'll definitely dig a bit deeper around this.

On Thu, Apr 21, 2016 at 10:18 AM, Stephen Mallette <sp...@gmail.com>
wrote:

> I'm not sure what you mean by:
>
> > Why is this not an option: It boils down to the fact that you still
> can't modify
> a traversal "after the fact".
>
> You can modify the Traversal up to the point where you iterate it and
> TraversalStrategy are applied.
>
> gremlin> base = g.V().hasLabel('person').out();[]
> gremlin> filter = __.has('age',29)
> gremlin> TraversalHelper.insertTraversal(1,filter,base);[]
> gremlin> base.toString()
> ==>[GraphStep(vertex,[]), HasStep([~label.eq(person)]),
> HasStep([age.eq(29)]), VertexStep(OUT,vertex)]
> gremlin> base
> ==>v[3]
> ==>v[2]
> ==>v[4]
>
> Note how I injected an anonymous Traversal (i.e. the "filter") into the
> middle of a "base" Traversal. This is a little known function that is
> probably familiar to TraversalStrategy developers only, but it exists. I
> suppose more advanced forms of Gremlin Language Variants should/could have
> this kind of capability - I guess the point is that it's possible and
> arguably better than string manipulation. You might not agree, but am I at
> least grasping the problem you're trying to express?
>
>
> On Thu, Apr 21, 2016 at 9:49 AM, Dylan Millikin <dy...@gmail.com>
> wrote:
>
> > Essentially what is required is to be able to "build" traversals with a
> > complete disregard for the order of the steps you're adding. Think of it
> as
> > adding entries to a match(). It doesn't matter what order you add those
> > entries in.
> > This is something that is very "declarative" in nature but is a
> requirement
> > when developing complex applications that need modularity. You also need
> > this in an imperative setting as you may want control on how the data is
> > fetched.
> >
> > On Thu, Apr 21, 2016 at 9:39 AM, Dylan Millikin <
> dylan.millikin@gmail.com>
> > wrote:
> >
> > > Yeah there are some more complex situations. You actually left out the
> > > final out() in your example. For instance just keeping things simple,
> > > your base query could be required to be reused in many places. So the
> > > following is not an option:
> > >
> > > base = g.V().hasLabel('person');[]
> > > /** apply some filters like you did **/
> > > base.out('company')
> > >
> > > Why is this not an option: It boils down to the fact that you still
> can't
> > > modify a traversal "after the fact".
> > > Your reusable base is not complete without the out() step. Therefore,
> > > using this model you can't store the base query and test it separately.
> > You
> > > also have to append the out step everywhere you use this base, which in
> > > addition to creating a lot of duplicate code can also not be an option
> > when
> > > queries are generated automatically. Sometimes you don't even really
> know
> > > what the base query is, you just know that it contains certain
> "injection
> > > points".
> > > To give you a few examples, these following base queries all have the
> > same
> > > injection point that allows them to be edited with the same filters:
> > >
> > > g.V().hasLabel("person")/** inject here **/.out('company');
> > > g.V().hasLabel("person")/** inject here
> **/.out('friend').out('company');
> > > g.V().has("person", "name", "marko").out('friend')/** inject here
> > > **/.out('company');
> > >
> > > These are all valid. My filters apply to "people" but the traversals
> > > leading to and from these people can be anything. And programmatically
> I
> > > have no way of knowing what the traversal is. This gets increasingly
> > > complexe with injection points in sub-traversals, or when injecting
> full
> > > traversals with their own injection points, and so on.
> > >
> > > Also your example doesn't cover changing existing steps.
> > >
> > > I don't know if that sheds anymore light.
> > >
> > > On Thu, Apr 21, 2016 at 9:14 AM, Stephen Mallette <
> spmallette@gmail.com>
> > > wrote:
> > >
> > >> So "query building" is where you hold string representations of steps
> or
> > >> groups of steps and then have some logic that concatenates them
> > together.
> > >> I
> > >> guess my question is why are "strings" a requirement for that? It
> seems
> > >> like you could do the same query builder stuff with an actual
> Traversal
> > >> object in whatever Gremlin Language Variant you were working with. How
> > is
> > >> string concatenation different than this:
> > >>
> > >> gremlin> graph = TinkerFactory.createModern()
> > >> ==>tinkergraph[vertices:6 edges:6]
> > >> gremlin> g = graph.traversal()
> > >> ==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
> > >> gremlin> base = g.V().hasLabel('person');[]
> > >> gremlin> addFilters = { age, name, t ->
> > >> gremlin>   t = age > 0 ? t.has('age',age) : t
> > >> gremlin>   !name.isEmpty() ? t.has('name',name) : t
> > >> gremlin> }
> > >> ==>groovysh_evaluate$_run_closure1@503d56b5
> > >> gremlin> traversal = addFilters(29,'',base);[]
> > >> gremlin> traversal.toString()
> > >> ==>[GraphStep(vertex,[]), HasStep([~label.eq(person)]),
> > >> HasStep([age.eq(29)])]
> > >> gremlin> traversal
> > >> ==>v[1]
> > >>
> > >> What about DSLs? Maybe filtering logic drops behind a custom step
> > specific
> > >> to the domain:
> > >>
> > >> g.V().personsWith(29, '')
> > >>
> > >> which would basically compile to the same thing as the gremlin output
> > >> above:
> > >>
> > >> [GraphStep(vertex,[]), HasStep([~label.eq(person)]),
> > >> HasStep([age.eq(29)])]
> > >>
> > >> Is there some more complex aspect "query building" that can't be
> handled
> > >> this way (I know I took a fairly simple example)?
> > >>
> > >>
> > >>
> > >> On Wed, Apr 20, 2016 at 7:54 PM, Dylan Millikin <
> > dylan.millikin@gmail.com
> > >> >
> > >> wrote:
> > >>
> > >> > I've approached this a few times in the past though never really in
> > >> depth.
> > >> > The idea is that you want separation of logic in your queries,
> mostly
> > >> for
> > >> > maintenance, testing and overall convenience. This takes the form of
> > >> > partial/incomplete traversals that have no start nor end.
> > >> > One of the simplest applications would be to have a base query and
> > apply
> > >> > filters to this query. For example we can imagine a recap page of
> > >> > everyone's companies:
> > >> >
> > >> > base query : g.V().has(label, "person").out('company')
> > >> >
> > >> > To this you have a set of filters that are partial traversals and
> > allow
> > >> > users to better refine their search:
> > >> >
> > >> > older than 30 filter: has("age", gt(30))
> > >> > male filter: has("gender", "male")
> > >> >
> > >> > Now depending on user input you'll want to apply either or both of
> the
> > >> > above to "person" in order to obtain something along the lines of:
> > >> >
> > >> > g.V().has(label, "person").has("age", gt(30)).has("gender",
> > >> > "male").out('company')
> > >> >
> > >> > Of course there are plenty of ways of doing the above depending on
> > >> > requirements and complexity. Such as where() or
> as("a")....select("a")
> > >> > etc..
> > >> >
> > >> > These are simple examples where steps are appended in various places
> > but
> > >> > you can imagine the same with traversal manipulation such as turning
> > >> > out('company') into out('company', 'organization') where the step
> gets
> > >> > altered. This would work the same way if you used
> > union(out('company'))
> > >> > instead. You would have to alter the union step like this :
> > >> > union(out('company'),
> > >> > out('organization'))
> > >> >
> > >> > All of the above are relatively simple but picture having a lot of
> > very
> > >> > complex traversal "filters" that sometimes have dependencies on
> other
> > >> > filters.
> > >> > All this is doable but it currently requires looking at the gremlin
> > >> > building process as a query building one rather than native support
> > for
> > >> the
> > >> > gremlin language.
> > >> >
> > >> > Does that make any sense?
> > >> >
> > >> > On Wed, Apr 20, 2016 at 7:15 PM, Stephen Mallette <
> > spmallette@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > >  It's imperative that you have the ability to group multiline
> > >> scripts
> > >> > > into a single query or the reactivity of your applications will
> > >> greatly
> > >> > > suffer.
> > >> > >
> > >> > > A fair point and something we might yet address as part of all
> this
> > >> > > thinking. RemoteGraph is pretty new and it demonstrated a critical
> > >> aspect
> > >> > > of communications with Gremlin Server. Now we need to think about
> > how
> > >> to
> > >> > > improve upon it.
> > >> > >
> > >> > > > Namely having the ability to edit/modify/add steps to an already
> > >> > defined
> > >> > > traversal.
> > >> > >
> > >> > > I don't quite follow that point...could you please elaborate?
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Apr 20, 2016 at 4:15 PM, Dylan Millikin <
> > >> > dylan.millikin@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Nice post,
> > >> > > >
> > >> > > > I'm going to jump straight to the end. I think that one of the
> > >> problems
> > >> > > of
> > >> > > > doing an abstraction similar to remoteGraph is that in reality
> the
> > >> > > overhead
> > >> > > > of communicating with the server is too big to make this viable.
> > >> It's
> > >> > > > imperative that you have the ability to group multiline scripts
> > >> into a
> > >> > > > single query or the reactivity of your applications will greatly
> > >> > suffer.
> > >> > > >
> > >> > > > The Gremlin-java base also has a few limitations that some query
> > >> > builders
> > >> > > > try to fix (which can only be done if you abandon the idea of a
> > >> natural
> > >> > > > gremlin language variant in favor of a query builder). Namely
> > having
> > >> > the
> > >> > > > ability to edit/modify/add steps to an already defined
> traversal.
> > >> > Though
> > >> > > in
> > >> > > > time it might be nice to have these be part of the original
> > >> > > implementation.
> > >> > > >
> > >> > > > I personally love the idea of gremlin language variants. I just
> > >> don't
> > >> > > think
> > >> > > > they're production value is any good without some extended
> > >> > functionality
> > >> > > > (beyond what gremlin-java currently is).
> > >> > > >
> > >> > > > On Wed, Apr 20, 2016 at 3:23 PM, Stephen Mallette <
> > >> > spmallette@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > This thread on Gremlin Language Variants has been very
> > >> interesting:
> > >> > > > >
> > >> > > > > https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
> > >> > > > >
> > >> > > > > I think that this work goes a long way to address two issues
> > I've
> > >> > been
> > >> > > > > concerned about:
> > >> > > > >
> > >> > > > > 1. Greater consistency in how different languages do Gremlin
> > >> > > > > 2. Less fragmentation in terms of libraries and how they work
> so
> > >> that
> > >> > > > users
> > >> > > > > aren't confused with how to get started (though I don't think
> > the
> > >> > goal
> > >> > > > here
> > >> > > > > is to restrict choices or slow down innovation)
> > >> > > > >
> > >> > > > > One of the first things we should probably do is start
> thinking
> > in
> > >> > > terms
> > >> > > > of
> > >> > > > > the types of libraries that are built on TinkerPop (outside of
> > >> those
> > >> > > > things
> > >> > > > > that are Graph Systems) and those are listed here currently:
> > >> > > > >
> > >> > > > > http://tinkerpop.apache.org/#graph-libraries
> > >> > > > >
> > >> > > > > Marko mentioned to me that he saw the libraries we listed here
> > >> > breaking
> > >> > > > > into three categories:
> > >> > > > >
> > >> > > > > 1. Gremlin Language Variants - which the other thread
> > demonstrates
> > >> > > quite
> > >> > > > > nicely
> > >> > > > > 2. Gremlin Drivers - the Gremlin Server protocol
> > implementations -
> > >> > > those
> > >> > > > > things that send traversals to Gremlin Server and get back
> > >> results.
> > >> > > > > 3. OGM and others - I say "others" because there might be
> > plugins
> > >> and
> > >> > > > other
> > >> > > > > similar odds and ends
> > >> > > > >
> > >> > > > > I like Marko's category system here and I think that having
> > these
> > >> > kinds
> > >> > > > of
> > >> > > > > categories will help folks organize their libraries to fit
> into
> > >> one
> > >> > of
> > >> > > > > these spaces and make it easier for users to know what they
> need
> > >> to
> > >> > get
> > >> > > > in
> > >> > > > > order to start doing TinkerPop in their language.
> > >> > > > >
> > >> > > > > Anyway, the category thing is just setting the stage for this
> > big
> > >> > > > > bombshell.  I think TinkerPop should consider maintaining the
> > >> Gremlin
> > >> > > > > Language Variants.
> > >> > > > >
> > >> > > > > Heresy! right?
> > >> > > > >
> > >> > > > > Well, I think it's the best way to achieve consistency across
> > >> > > languages.
> > >> > > > > Under this model, TinkerPop provides the base language variant
> > and
> > >> > > people
> > >> > > > > can choose to extend upon it, but the base stays tied to our
> > >> > archetype
> > >> > > of
> > >> > > > > Java and we end up with a much more clear story for virtually
> > any
> > >> > > > > programming language.
> > >> > > > >
> > >> > > > > So how do we do this? Slowly and deliberately. We should look
> to
> > >> only
> > >> > > > > include language variants where we:
> > >> > > > >
> > >> > > > > + have good automation in place (like what Marko did for
> > Python),
> > >> > > > > + some competence on the committer list in that language
> > >> > > > > + a nice testing framework that operates in our standard
> > >> > build/release
> > >> > > > > process.
> > >> > > > >
> > >> > > > > That's setting a high bar, but if we don't keep it high, I
> think
> > >> we
> > >> > > will
> > >> > > > be
> > >> > > > > left unable to properly support and maintain what we hang out
> > >> there.
> > >> > > > >
> > >> > > > > I'd also like to express that we should not be looking to
> > maintain
> > >> > > > language
> > >> > > > > drivers. I think that should remain a third-party community
> > effort
> > >> > just
> > >> > > > > like Graph Systems. In other words, we remain a repository for
> > >> > > reference
> > >> > > > > implementations for everything else. Why? Because, as it sits
> > >> right
> > >> > > now,
> > >> > > > > just based on the level of effort for what Marko did with
> > Python,
> > >> > > > > maintaining a "base" Gremlin Language Variants shouldn't be
> > hard.
> > >> We
> > >> > > > won't
> > >> > > > > be building tons of add-on capabilities to the base variants -
> > >> they
> > >> > > will
> > >> > > > > pretty much just stay on par with the java archetype.  Drivers
> > on
> > >> the
> > >> > > > other
> > >> > > > > hand have lots of implementation details, with many different
> > >> > > > technologies
> > >> > > > > that could be used, etc.  They have similar complexity to
> Graph
> > >> > System
> > >> > > > > implementations in many ways. I also think that the drivers
> can
> > >> > afford
> > >> > > to
> > >> > > > > have different APIs and approaches without being detrimental
> to
> > >> the
> > >> > > > > community.
> > >> > > > >
> > >> > > > > If gremlin-js-driver wants to do:
> > >> > > > >
> > >> > > > > client.submit("g.V()")
> > >> > > > >
> > >> > > > > and gremlin-python-driver wants to do:
> > >> > > > >
> > >> > > > > client.send("g.V()")
> > >> > > > >
> > >> > > > > that's not a big deal.
> > >> > > > >
> > >> > > > > The last point that I'll make is that I think Gremlin Language
> > >> > > Variants,
> > >> > > > > that don't operate on the JVM (e.g. Jython) and use Gremlin
> > >> Server,
> > >> > > > should
> > >> > > > > have some abstraction that is similar to RemoteGraph.
> > RemoteGraph
> > >> > > > exposes
> > >> > > > > a DriverConnection interface that is currently implemented by
> > >> > > > > gremlin-driver.  The DriverConnection is responsible for
> > sending a
> > >> > > > > traversal to the server and returning results. It would be
> nice
> > if
> > >> > the
> > >> > > > > language variants had a similar interface that the various
> > >> community
> > >> > > > > drivers could implement. In that way, the user never has to do
> > any
> > >> > form
> > >> > > > of:
> > >> > > > >
> > >> > > > > client.submit(someGremlinString)
> > >> > > > >
> > >> > > > > in any language. We really need to try to make that pattern go
> > >> away
> > >> > > > across
> > >> > > > > the TinkerPop community.
> > >> > > > >
> > >> > > > > So - that's was a long email. Looking forward to hearing some
> > >> > > discussion
> > >> > > > on
> > >> > > > > this.
> > >> > > > >
> > >> > > > > Stephen
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Stephen Mallette <sp...@gmail.com>]

I'm not sure what you mean by:

> Why is this not an option: It boils down to the fact that you still can't modify
a traversal "after the fact".

You can modify the Traversal up to the point where you iterate it and
TraversalStrategy are applied.

gremlin> base = g.V().hasLabel('person').out();[]
gremlin> filter = __.has('age',29)
gremlin> TraversalHelper.insertTraversal(1,filter,base);[]
gremlin> base.toString()
==>[GraphStep(vertex,[]), HasStep([~label.eq(person)]),
HasStep([age.eq(29)]), VertexStep(OUT,vertex)]
gremlin> base
==>v[3]
==>v[2]
==>v[4]

Note how I injected an anonymous Traversal (i.e. the "filter") into the
middle of a "base" Traversal. This is a little known function that is
probably familiar to TraversalStrategy developers only, but it exists. I
suppose more advanced forms of Gremlin Language Variants should/could have
this kind of capability - I guess the point is that it's possible and
arguably better than string manipulation. You might not agree, but am I at
least grasping the problem you're trying to express?


On Thu, Apr 21, 2016 at 9:49 AM, Dylan Millikin <dy...@gmail.com>
wrote:

> Essentially what is required is to be able to "build" traversals with a
> complete disregard for the order of the steps you're adding. Think of it as
> adding entries to a match(). It doesn't matter what order you add those
> entries in.
> This is something that is very "declarative" in nature but is a requirement
> when developing complex applications that need modularity. You also need
> this in an imperative setting as you may want control on how the data is
> fetched.
>
> On Thu, Apr 21, 2016 at 9:39 AM, Dylan Millikin <dy...@gmail.com>
> wrote:
>
> > Yeah there are some more complex situations. You actually left out the
> > final out() in your example. For instance just keeping things simple,
> > your base query could be required to be reused in many places. So the
> > following is not an option:
> >
> > base = g.V().hasLabel('person');[]
> > /** apply some filters like you did **/
> > base.out('company')
> >
> > Why is this not an option: It boils down to the fact that you still can't
> > modify a traversal "after the fact".
> > Your reusable base is not complete without the out() step. Therefore,
> > using this model you can't store the base query and test it separately.
> You
> > also have to append the out step everywhere you use this base, which in
> > addition to creating a lot of duplicate code can also not be an option
> when
> > queries are generated automatically. Sometimes you don't even really know
> > what the base query is, you just know that it contains certain "injection
> > points".
> > To give you a few examples, these following base queries all have the
> same
> > injection point that allows them to be edited with the same filters:
> >
> > g.V().hasLabel("person")/** inject here **/.out('company');
> > g.V().hasLabel("person")/** inject here **/.out('friend').out('company');
> > g.V().has("person", "name", "marko").out('friend')/** inject here
> > **/.out('company');
> >
> > These are all valid. My filters apply to "people" but the traversals
> > leading to and from these people can be anything. And programmatically I
> > have no way of knowing what the traversal is. This gets increasingly
> > complexe with injection points in sub-traversals, or when injecting full
> > traversals with their own injection points, and so on.
> >
> > Also your example doesn't cover changing existing steps.
> >
> > I don't know if that sheds anymore light.
> >
> > On Thu, Apr 21, 2016 at 9:14 AM, Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> >> So "query building" is where you hold string representations of steps or
> >> groups of steps and then have some logic that concatenates them
> together.
> >> I
> >> guess my question is why are "strings" a requirement for that? It seems
> >> like you could do the same query builder stuff with an actual Traversal
> >> object in whatever Gremlin Language Variant you were working with. How
> is
> >> string concatenation different than this:
> >>
> >> gremlin> graph = TinkerFactory.createModern()
> >> ==>tinkergraph[vertices:6 edges:6]
> >> gremlin> g = graph.traversal()
> >> ==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
> >> gremlin> base = g.V().hasLabel('person');[]
> >> gremlin> addFilters = { age, name, t ->
> >> gremlin>   t = age > 0 ? t.has('age',age) : t
> >> gremlin>   !name.isEmpty() ? t.has('name',name) : t
> >> gremlin> }
> >> ==>groovysh_evaluate$_run_closure1@503d56b5
> >> gremlin> traversal = addFilters(29,'',base);[]
> >> gremlin> traversal.toString()
> >> ==>[GraphStep(vertex,[]), HasStep([~label.eq(person)]),
> >> HasStep([age.eq(29)])]
> >> gremlin> traversal
> >> ==>v[1]
> >>
> >> What about DSLs? Maybe filtering logic drops behind a custom step
> specific
> >> to the domain:
> >>
> >> g.V().personsWith(29, '')
> >>
> >> which would basically compile to the same thing as the gremlin output
> >> above:
> >>
> >> [GraphStep(vertex,[]), HasStep([~label.eq(person)]),
> >> HasStep([age.eq(29)])]
> >>
> >> Is there some more complex aspect "query building" that can't be handled
> >> this way (I know I took a fairly simple example)?
> >>
> >>
> >>
> >> On Wed, Apr 20, 2016 at 7:54 PM, Dylan Millikin <
> dylan.millikin@gmail.com
> >> >
> >> wrote:
> >>
> >> > I've approached this a few times in the past though never really in
> >> depth.
> >> > The idea is that you want separation of logic in your queries, mostly
> >> for
> >> > maintenance, testing and overall convenience. This takes the form of
> >> > partial/incomplete traversals that have no start nor end.
> >> > One of the simplest applications would be to have a base query and
> apply
> >> > filters to this query. For example we can imagine a recap page of
> >> > everyone's companies:
> >> >
> >> > base query : g.V().has(label, "person").out('company')
> >> >
> >> > To this you have a set of filters that are partial traversals and
> allow
> >> > users to better refine their search:
> >> >
> >> > older than 30 filter: has("age", gt(30))
> >> > male filter: has("gender", "male")
> >> >
> >> > Now depending on user input you'll want to apply either or both of the
> >> > above to "person" in order to obtain something along the lines of:
> >> >
> >> > g.V().has(label, "person").has("age", gt(30)).has("gender",
> >> > "male").out('company')
> >> >
> >> > Of course there are plenty of ways of doing the above depending on
> >> > requirements and complexity. Such as where() or as("a")....select("a")
> >> > etc..
> >> >
> >> > These are simple examples where steps are appended in various places
> but
> >> > you can imagine the same with traversal manipulation such as turning
> >> > out('company') into out('company', 'organization') where the step gets
> >> > altered. This would work the same way if you used
> union(out('company'))
> >> > instead. You would have to alter the union step like this :
> >> > union(out('company'),
> >> > out('organization'))
> >> >
> >> > All of the above are relatively simple but picture having a lot of
> very
> >> > complex traversal "filters" that sometimes have dependencies on other
> >> > filters.
> >> > All this is doable but it currently requires looking at the gremlin
> >> > building process as a query building one rather than native support
> for
> >> the
> >> > gremlin language.
> >> >
> >> > Does that make any sense?
> >> >
> >> > On Wed, Apr 20, 2016 at 7:15 PM, Stephen Mallette <
> spmallette@gmail.com
> >> >
> >> > wrote:
> >> >
> >> > > >  It's imperative that you have the ability to group multiline
> >> scripts
> >> > > into a single query or the reactivity of your applications will
> >> greatly
> >> > > suffer.
> >> > >
> >> > > A fair point and something we might yet address as part of all this
> >> > > thinking. RemoteGraph is pretty new and it demonstrated a critical
> >> aspect
> >> > > of communications with Gremlin Server. Now we need to think about
> how
> >> to
> >> > > improve upon it.
> >> > >
> >> > > > Namely having the ability to edit/modify/add steps to an already
> >> > defined
> >> > > traversal.
> >> > >
> >> > > I don't quite follow that point...could you please elaborate?
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Apr 20, 2016 at 4:15 PM, Dylan Millikin <
> >> > dylan.millikin@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Nice post,
> >> > > >
> >> > > > I'm going to jump straight to the end. I think that one of the
> >> problems
> >> > > of
> >> > > > doing an abstraction similar to remoteGraph is that in reality the
> >> > > overhead
> >> > > > of communicating with the server is too big to make this viable.
> >> It's
> >> > > > imperative that you have the ability to group multiline scripts
> >> into a
> >> > > > single query or the reactivity of your applications will greatly
> >> > suffer.
> >> > > >
> >> > > > The Gremlin-java base also has a few limitations that some query
> >> > builders
> >> > > > try to fix (which can only be done if you abandon the idea of a
> >> natural
> >> > > > gremlin language variant in favor of a query builder). Namely
> having
> >> > the
> >> > > > ability to edit/modify/add steps to an already defined traversal.
> >> > Though
> >> > > in
> >> > > > time it might be nice to have these be part of the original
> >> > > implementation.
> >> > > >
> >> > > > I personally love the idea of gremlin language variants. I just
> >> don't
> >> > > think
> >> > > > they're production value is any good without some extended
> >> > functionality
> >> > > > (beyond what gremlin-java currently is).
> >> > > >
> >> > > > On Wed, Apr 20, 2016 at 3:23 PM, Stephen Mallette <
> >> > spmallette@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > This thread on Gremlin Language Variants has been very
> >> interesting:
> >> > > > >
> >> > > > > https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
> >> > > > >
> >> > > > > I think that this work goes a long way to address two issues
> I've
> >> > been
> >> > > > > concerned about:
> >> > > > >
> >> > > > > 1. Greater consistency in how different languages do Gremlin
> >> > > > > 2. Less fragmentation in terms of libraries and how they work so
> >> that
> >> > > > users
> >> > > > > aren't confused with how to get started (though I don't think
> the
> >> > goal
> >> > > > here
> >> > > > > is to restrict choices or slow down innovation)
> >> > > > >
> >> > > > > One of the first things we should probably do is start thinking
> in
> >> > > terms
> >> > > > of
> >> > > > > the types of libraries that are built on TinkerPop (outside of
> >> those
> >> > > > things
> >> > > > > that are Graph Systems) and those are listed here currently:
> >> > > > >
> >> > > > > http://tinkerpop.apache.org/#graph-libraries
> >> > > > >
> >> > > > > Marko mentioned to me that he saw the libraries we listed here
> >> > breaking
> >> > > > > into three categories:
> >> > > > >
> >> > > > > 1. Gremlin Language Variants - which the other thread
> demonstrates
> >> > > quite
> >> > > > > nicely
> >> > > > > 2. Gremlin Drivers - the Gremlin Server protocol
> implementations -
> >> > > those
> >> > > > > things that send traversals to Gremlin Server and get back
> >> results.
> >> > > > > 3. OGM and others - I say "others" because there might be
> plugins
> >> and
> >> > > > other
> >> > > > > similar odds and ends
> >> > > > >
> >> > > > > I like Marko's category system here and I think that having
> these
> >> > kinds
> >> > > > of
> >> > > > > categories will help folks organize their libraries to fit into
> >> one
> >> > of
> >> > > > > these spaces and make it easier for users to know what they need
> >> to
> >> > get
> >> > > > in
> >> > > > > order to start doing TinkerPop in their language.
> >> > > > >
> >> > > > > Anyway, the category thing is just setting the stage for this
> big
> >> > > > > bombshell.  I think TinkerPop should consider maintaining the
> >> Gremlin
> >> > > > > Language Variants.
> >> > > > >
> >> > > > > Heresy! right?
> >> > > > >
> >> > > > > Well, I think it's the best way to achieve consistency across
> >> > > languages.
> >> > > > > Under this model, TinkerPop provides the base language variant
> and
> >> > > people
> >> > > > > can choose to extend upon it, but the base stays tied to our
> >> > archetype
> >> > > of
> >> > > > > Java and we end up with a much more clear story for virtually
> any
> >> > > > > programming language.
> >> > > > >
> >> > > > > So how do we do this? Slowly and deliberately. We should look to
> >> only
> >> > > > > include language variants where we:
> >> > > > >
> >> > > > > + have good automation in place (like what Marko did for
> Python),
> >> > > > > + some competence on the committer list in that language
> >> > > > > + a nice testing framework that operates in our standard
> >> > build/release
> >> > > > > process.
> >> > > > >
> >> > > > > That's setting a high bar, but if we don't keep it high, I think
> >> we
> >> > > will
> >> > > > be
> >> > > > > left unable to properly support and maintain what we hang out
> >> there.
> >> > > > >
> >> > > > > I'd also like to express that we should not be looking to
> maintain
> >> > > > language
> >> > > > > drivers. I think that should remain a third-party community
> effort
> >> > just
> >> > > > > like Graph Systems. In other words, we remain a repository for
> >> > > reference
> >> > > > > implementations for everything else. Why? Because, as it sits
> >> right
> >> > > now,
> >> > > > > just based on the level of effort for what Marko did with
> Python,
> >> > > > > maintaining a "base" Gremlin Language Variants shouldn't be
> hard.
> >> We
> >> > > > won't
> >> > > > > be building tons of add-on capabilities to the base variants -
> >> they
> >> > > will
> >> > > > > pretty much just stay on par with the java archetype.  Drivers
> on
> >> the
> >> > > > other
> >> > > > > hand have lots of implementation details, with many different
> >> > > > technologies
> >> > > > > that could be used, etc.  They have similar complexity to Graph
> >> > System
> >> > > > > implementations in many ways. I also think that the drivers can
> >> > afford
> >> > > to
> >> > > > > have different APIs and approaches without being detrimental to
> >> the
> >> > > > > community.
> >> > > > >
> >> > > > > If gremlin-js-driver wants to do:
> >> > > > >
> >> > > > > client.submit("g.V()")
> >> > > > >
> >> > > > > and gremlin-python-driver wants to do:
> >> > > > >
> >> > > > > client.send("g.V()")
> >> > > > >
> >> > > > > that's not a big deal.
> >> > > > >
> >> > > > > The last point that I'll make is that I think Gremlin Language
> >> > > Variants,
> >> > > > > that don't operate on the JVM (e.g. Jython) and use Gremlin
> >> Server,
> >> > > > should
> >> > > > > have some abstraction that is similar to RemoteGraph.
> RemoteGraph
> >> > > > exposes
> >> > > > > a DriverConnection interface that is currently implemented by
> >> > > > > gremlin-driver.  The DriverConnection is responsible for
> sending a
> >> > > > > traversal to the server and returning results. It would be nice
> if
> >> > the
> >> > > > > language variants had a similar interface that the various
> >> community
> >> > > > > drivers could implement. In that way, the user never has to do
> any
> >> > form
> >> > > > of:
> >> > > > >
> >> > > > > client.submit(someGremlinString)
> >> > > > >
> >> > > > > in any language. We really need to try to make that pattern go
> >> away
> >> > > > across
> >> > > > > the TinkerPop community.
> >> > > > >
> >> > > > > So - that's was a long email. Looking forward to hearing some
> >> > > discussion
> >> > > > on
> >> > > > > this.
> >> > > > >
> >> > > > > Stephen
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Dylan Millikin <dy...@gmail.com>]

Essentially what is required is to be able to "build" traversals with a
complete disregard for the order of the steps you're adding. Think of it as
adding entries to a match(). It doesn't matter what order you add those
entries in.
This is something that is very "declarative" in nature but is a requirement
when developing complex applications that need modularity. You also need
this in an imperative setting as you may want control on how the data is
fetched.

On Thu, Apr 21, 2016 at 9:39 AM, Dylan Millikin <dy...@gmail.com>
wrote:

> Yeah there are some more complex situations. You actually left out the
> final out() in your example. For instance just keeping things simple,
> your base query could be required to be reused in many places. So the
> following is not an option:
>
> base = g.V().hasLabel('person');[]
> /** apply some filters like you did **/
> base.out('company')
>
> Why is this not an option: It boils down to the fact that you still can't
> modify a traversal "after the fact".
> Your reusable base is not complete without the out() step. Therefore,
> using this model you can't store the base query and test it separately. You
> also have to append the out step everywhere you use this base, which in
> addition to creating a lot of duplicate code can also not be an option when
> queries are generated automatically. Sometimes you don't even really know
> what the base query is, you just know that it contains certain "injection
> points".
> To give you a few examples, these following base queries all have the same
> injection point that allows them to be edited with the same filters:
>
> g.V().hasLabel("person")/** inject here **/.out('company');
> g.V().hasLabel("person")/** inject here **/.out('friend').out('company');
> g.V().has("person", "name", "marko").out('friend')/** inject here
> **/.out('company');
>
> These are all valid. My filters apply to "people" but the traversals
> leading to and from these people can be anything. And programmatically I
> have no way of knowing what the traversal is. This gets increasingly
> complexe with injection points in sub-traversals, or when injecting full
> traversals with their own injection points, and so on.
>
> Also your example doesn't cover changing existing steps.
>
> I don't know if that sheds anymore light.
>
> On Thu, Apr 21, 2016 at 9:14 AM, Stephen Mallette <sp...@gmail.com>
> wrote:
>
>> So "query building" is where you hold string representations of steps or
>> groups of steps and then have some logic that concatenates them together.
>> I
>> guess my question is why are "strings" a requirement for that? It seems
>> like you could do the same query builder stuff with an actual Traversal
>> object in whatever Gremlin Language Variant you were working with. How is
>> string concatenation different than this:
>>
>> gremlin> graph = TinkerFactory.createModern()
>> ==>tinkergraph[vertices:6 edges:6]
>> gremlin> g = graph.traversal()
>> ==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
>> gremlin> base = g.V().hasLabel('person');[]
>> gremlin> addFilters = { age, name, t ->
>> gremlin>   t = age > 0 ? t.has('age',age) : t
>> gremlin>   !name.isEmpty() ? t.has('name',name) : t
>> gremlin> }
>> ==>groovysh_evaluate$_run_closure1@503d56b5
>> gremlin> traversal = addFilters(29,'',base);[]
>> gremlin> traversal.toString()
>> ==>[GraphStep(vertex,[]), HasStep([~label.eq(person)]),
>> HasStep([age.eq(29)])]
>> gremlin> traversal
>> ==>v[1]
>>
>> What about DSLs? Maybe filtering logic drops behind a custom step specific
>> to the domain:
>>
>> g.V().personsWith(29, '')
>>
>> which would basically compile to the same thing as the gremlin output
>> above:
>>
>> [GraphStep(vertex,[]), HasStep([~label.eq(person)]),
>> HasStep([age.eq(29)])]
>>
>> Is there some more complex aspect "query building" that can't be handled
>> this way (I know I took a fairly simple example)?
>>
>>
>>
>> On Wed, Apr 20, 2016 at 7:54 PM, Dylan Millikin <dylan.millikin@gmail.com
>> >
>> wrote:
>>
>> > I've approached this a few times in the past though never really in
>> depth.
>> > The idea is that you want separation of logic in your queries, mostly
>> for
>> > maintenance, testing and overall convenience. This takes the form of
>> > partial/incomplete traversals that have no start nor end.
>> > One of the simplest applications would be to have a base query and apply
>> > filters to this query. For example we can imagine a recap page of
>> > everyone's companies:
>> >
>> > base query : g.V().has(label, "person").out('company')
>> >
>> > To this you have a set of filters that are partial traversals and allow
>> > users to better refine their search:
>> >
>> > older than 30 filter: has("age", gt(30))
>> > male filter: has("gender", "male")
>> >
>> > Now depending on user input you'll want to apply either or both of the
>> > above to "person" in order to obtain something along the lines of:
>> >
>> > g.V().has(label, "person").has("age", gt(30)).has("gender",
>> > "male").out('company')
>> >
>> > Of course there are plenty of ways of doing the above depending on
>> > requirements and complexity. Such as where() or as("a")....select("a")
>> > etc..
>> >
>> > These are simple examples where steps are appended in various places but
>> > you can imagine the same with traversal manipulation such as turning
>> > out('company') into out('company', 'organization') where the step gets
>> > altered. This would work the same way if you used union(out('company'))
>> > instead. You would have to alter the union step like this :
>> > union(out('company'),
>> > out('organization'))
>> >
>> > All of the above are relatively simple but picture having a lot of very
>> > complex traversal "filters" that sometimes have dependencies on other
>> > filters.
>> > All this is doable but it currently requires looking at the gremlin
>> > building process as a query building one rather than native support for
>> the
>> > gremlin language.
>> >
>> > Does that make any sense?
>> >
>> > On Wed, Apr 20, 2016 at 7:15 PM, Stephen Mallette <spmallette@gmail.com
>> >
>> > wrote:
>> >
>> > > >  It's imperative that you have the ability to group multiline
>> scripts
>> > > into a single query or the reactivity of your applications will
>> greatly
>> > > suffer.
>> > >
>> > > A fair point and something we might yet address as part of all this
>> > > thinking. RemoteGraph is pretty new and it demonstrated a critical
>> aspect
>> > > of communications with Gremlin Server. Now we need to think about how
>> to
>> > > improve upon it.
>> > >
>> > > > Namely having the ability to edit/modify/add steps to an already
>> > defined
>> > > traversal.
>> > >
>> > > I don't quite follow that point...could you please elaborate?
>> > >
>> > >
>> > >
>> > >
>> > > On Wed, Apr 20, 2016 at 4:15 PM, Dylan Millikin <
>> > dylan.millikin@gmail.com>
>> > > wrote:
>> > >
>> > > > Nice post,
>> > > >
>> > > > I'm going to jump straight to the end. I think that one of the
>> problems
>> > > of
>> > > > doing an abstraction similar to remoteGraph is that in reality the
>> > > overhead
>> > > > of communicating with the server is too big to make this viable.
>> It's
>> > > > imperative that you have the ability to group multiline scripts
>> into a
>> > > > single query or the reactivity of your applications will greatly
>> > suffer.
>> > > >
>> > > > The Gremlin-java base also has a few limitations that some query
>> > builders
>> > > > try to fix (which can only be done if you abandon the idea of a
>> natural
>> > > > gremlin language variant in favor of a query builder). Namely having
>> > the
>> > > > ability to edit/modify/add steps to an already defined traversal.
>> > Though
>> > > in
>> > > > time it might be nice to have these be part of the original
>> > > implementation.
>> > > >
>> > > > I personally love the idea of gremlin language variants. I just
>> don't
>> > > think
>> > > > they're production value is any good without some extended
>> > functionality
>> > > > (beyond what gremlin-java currently is).
>> > > >
>> > > > On Wed, Apr 20, 2016 at 3:23 PM, Stephen Mallette <
>> > spmallette@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > This thread on Gremlin Language Variants has been very
>> interesting:
>> > > > >
>> > > > > https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
>> > > > >
>> > > > > I think that this work goes a long way to address two issues I've
>> > been
>> > > > > concerned about:
>> > > > >
>> > > > > 1. Greater consistency in how different languages do Gremlin
>> > > > > 2. Less fragmentation in terms of libraries and how they work so
>> that
>> > > > users
>> > > > > aren't confused with how to get started (though I don't think the
>> > goal
>> > > > here
>> > > > > is to restrict choices or slow down innovation)
>> > > > >
>> > > > > One of the first things we should probably do is start thinking in
>> > > terms
>> > > > of
>> > > > > the types of libraries that are built on TinkerPop (outside of
>> those
>> > > > things
>> > > > > that are Graph Systems) and those are listed here currently:
>> > > > >
>> > > > > http://tinkerpop.apache.org/#graph-libraries
>> > > > >
>> > > > > Marko mentioned to me that he saw the libraries we listed here
>> > breaking
>> > > > > into three categories:
>> > > > >
>> > > > > 1. Gremlin Language Variants - which the other thread demonstrates
>> > > quite
>> > > > > nicely
>> > > > > 2. Gremlin Drivers - the Gremlin Server protocol implementations -
>> > > those
>> > > > > things that send traversals to Gremlin Server and get back
>> results.
>> > > > > 3. OGM and others - I say "others" because there might be plugins
>> and
>> > > > other
>> > > > > similar odds and ends
>> > > > >
>> > > > > I like Marko's category system here and I think that having these
>> > kinds
>> > > > of
>> > > > > categories will help folks organize their libraries to fit into
>> one
>> > of
>> > > > > these spaces and make it easier for users to know what they need
>> to
>> > get
>> > > > in
>> > > > > order to start doing TinkerPop in their language.
>> > > > >
>> > > > > Anyway, the category thing is just setting the stage for this big
>> > > > > bombshell.  I think TinkerPop should consider maintaining the
>> Gremlin
>> > > > > Language Variants.
>> > > > >
>> > > > > Heresy! right?
>> > > > >
>> > > > > Well, I think it's the best way to achieve consistency across
>> > > languages.
>> > > > > Under this model, TinkerPop provides the base language variant and
>> > > people
>> > > > > can choose to extend upon it, but the base stays tied to our
>> > archetype
>> > > of
>> > > > > Java and we end up with a much more clear story for virtually any
>> > > > > programming language.
>> > > > >
>> > > > > So how do we do this? Slowly and deliberately. We should look to
>> only
>> > > > > include language variants where we:
>> > > > >
>> > > > > + have good automation in place (like what Marko did for Python),
>> > > > > + some competence on the committer list in that language
>> > > > > + a nice testing framework that operates in our standard
>> > build/release
>> > > > > process.
>> > > > >
>> > > > > That's setting a high bar, but if we don't keep it high, I think
>> we
>> > > will
>> > > > be
>> > > > > left unable to properly support and maintain what we hang out
>> there.
>> > > > >
>> > > > > I'd also like to express that we should not be looking to maintain
>> > > > language
>> > > > > drivers. I think that should remain a third-party community effort
>> > just
>> > > > > like Graph Systems. In other words, we remain a repository for
>> > > reference
>> > > > > implementations for everything else. Why? Because, as it sits
>> right
>> > > now,
>> > > > > just based on the level of effort for what Marko did with Python,
>> > > > > maintaining a "base" Gremlin Language Variants shouldn't be hard.
>> We
>> > > > won't
>> > > > > be building tons of add-on capabilities to the base variants -
>> they
>> > > will
>> > > > > pretty much just stay on par with the java archetype.  Drivers on
>> the
>> > > > other
>> > > > > hand have lots of implementation details, with many different
>> > > > technologies
>> > > > > that could be used, etc.  They have similar complexity to Graph
>> > System
>> > > > > implementations in many ways. I also think that the drivers can
>> > afford
>> > > to
>> > > > > have different APIs and approaches without being detrimental to
>> the
>> > > > > community.
>> > > > >
>> > > > > If gremlin-js-driver wants to do:
>> > > > >
>> > > > > client.submit("g.V()")
>> > > > >
>> > > > > and gremlin-python-driver wants to do:
>> > > > >
>> > > > > client.send("g.V()")
>> > > > >
>> > > > > that's not a big deal.
>> > > > >
>> > > > > The last point that I'll make is that I think Gremlin Language
>> > > Variants,
>> > > > > that don't operate on the JVM (e.g. Jython) and use Gremlin
>> Server,
>> > > > should
>> > > > > have some abstraction that is similar to RemoteGraph.  RemoteGraph
>> > > > exposes
>> > > > > a DriverConnection interface that is currently implemented by
>> > > > > gremlin-driver.  The DriverConnection is responsible for sending a
>> > > > > traversal to the server and returning results. It would be nice if
>> > the
>> > > > > language variants had a similar interface that the various
>> community
>> > > > > drivers could implement. In that way, the user never has to do any
>> > form
>> > > > of:
>> > > > >
>> > > > > client.submit(someGremlinString)
>> > > > >
>> > > > > in any language. We really need to try to make that pattern go
>> away
>> > > > across
>> > > > > the TinkerPop community.
>> > > > >
>> > > > > So - that's was a long email. Looking forward to hearing some
>> > > discussion
>> > > > on
>> > > > > this.
>> > > > >
>> > > > > Stephen
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Dylan Millikin <dy...@gmail.com>]

Yeah there are some more complex situations. You actually left out the
final out() in your example. For instance just keeping things simple, your
base query could be required to be reused in many places. So the following
is not an option:

base = g.V().hasLabel('person');[]
/** apply some filters like you did **/
base.out('company')

Why is this not an option: It boils down to the fact that you still can't
modify a traversal "after the fact".
Your reusable base is not complete without the out() step. Therefore, using
this model you can't store the base query and test it separately. You also
have to append the out step everywhere you use this base, which in addition
to creating a lot of duplicate code can also not be an option when queries
are generated automatically. Sometimes you don't even really know what the
base query is, you just know that it contains certain "injection points".
To give you a few examples, these following base queries all have the same
injection point that allows them to be edited with the same filters:

g.V().hasLabel("person")/** inject here **/.out('company');
g.V().hasLabel("person")/** inject here **/.out('friend').out('company');
g.V().has("person", "name", "marko").out('friend')/** inject here
**/.out('company');

These are all valid. My filters apply to "people" but the traversals
leading to and from these people can be anything. And programmatically I
have no way of knowing what the traversal is. This gets increasingly
complexe with injection points in sub-traversals, or when injecting full
traversals with their own injection points, and so on.

Also your example doesn't cover changing existing steps.

I don't know if that sheds anymore light.

On Thu, Apr 21, 2016 at 9:14 AM, Stephen Mallette <sp...@gmail.com>
wrote:

> So "query building" is where you hold string representations of steps or
> groups of steps and then have some logic that concatenates them together. I
> guess my question is why are "strings" a requirement for that? It seems
> like you could do the same query builder stuff with an actual Traversal
> object in whatever Gremlin Language Variant you were working with. How is
> string concatenation different than this:
>
> gremlin> graph = TinkerFactory.createModern()
> ==>tinkergraph[vertices:6 edges:6]
> gremlin> g = graph.traversal()
> ==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
> gremlin> base = g.V().hasLabel('person');[]
> gremlin> addFilters = { age, name, t ->
> gremlin>   t = age > 0 ? t.has('age',age) : t
> gremlin>   !name.isEmpty() ? t.has('name',name) : t
> gremlin> }
> ==>groovysh_evaluate$_run_closure1@503d56b5
> gremlin> traversal = addFilters(29,'',base);[]
> gremlin> traversal.toString()
> ==>[GraphStep(vertex,[]), HasStep([~label.eq(person)]),
> HasStep([age.eq(29)])]
> gremlin> traversal
> ==>v[1]
>
> What about DSLs? Maybe filtering logic drops behind a custom step specific
> to the domain:
>
> g.V().personsWith(29, '')
>
> which would basically compile to the same thing as the gremlin output
> above:
>
> [GraphStep(vertex,[]), HasStep([~label.eq(person)]), HasStep([age.eq(29)])]
>
> Is there some more complex aspect "query building" that can't be handled
> this way (I know I took a fairly simple example)?
>
>
>
> On Wed, Apr 20, 2016 at 7:54 PM, Dylan Millikin <dy...@gmail.com>
> wrote:
>
> > I've approached this a few times in the past though never really in
> depth.
> > The idea is that you want separation of logic in your queries, mostly for
> > maintenance, testing and overall convenience. This takes the form of
> > partial/incomplete traversals that have no start nor end.
> > One of the simplest applications would be to have a base query and apply
> > filters to this query. For example we can imagine a recap page of
> > everyone's companies:
> >
> > base query : g.V().has(label, "person").out('company')
> >
> > To this you have a set of filters that are partial traversals and allow
> > users to better refine their search:
> >
> > older than 30 filter: has("age", gt(30))
> > male filter: has("gender", "male")
> >
> > Now depending on user input you'll want to apply either or both of the
> > above to "person" in order to obtain something along the lines of:
> >
> > g.V().has(label, "person").has("age", gt(30)).has("gender",
> > "male").out('company')
> >
> > Of course there are plenty of ways of doing the above depending on
> > requirements and complexity. Such as where() or as("a")....select("a")
> > etc..
> >
> > These are simple examples where steps are appended in various places but
> > you can imagine the same with traversal manipulation such as turning
> > out('company') into out('company', 'organization') where the step gets
> > altered. This would work the same way if you used union(out('company'))
> > instead. You would have to alter the union step like this :
> > union(out('company'),
> > out('organization'))
> >
> > All of the above are relatively simple but picture having a lot of very
> > complex traversal "filters" that sometimes have dependencies on other
> > filters.
> > All this is doable but it currently requires looking at the gremlin
> > building process as a query building one rather than native support for
> the
> > gremlin language.
> >
> > Does that make any sense?
> >
> > On Wed, Apr 20, 2016 at 7:15 PM, Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > >  It's imperative that you have the ability to group multiline scripts
> > > into a single query or the reactivity of your applications will greatly
> > > suffer.
> > >
> > > A fair point and something we might yet address as part of all this
> > > thinking. RemoteGraph is pretty new and it demonstrated a critical
> aspect
> > > of communications with Gremlin Server. Now we need to think about how
> to
> > > improve upon it.
> > >
> > > > Namely having the ability to edit/modify/add steps to an already
> > defined
> > > traversal.
> > >
> > > I don't quite follow that point...could you please elaborate?
> > >
> > >
> > >
> > >
> > > On Wed, Apr 20, 2016 at 4:15 PM, Dylan Millikin <
> > dylan.millikin@gmail.com>
> > > wrote:
> > >
> > > > Nice post,
> > > >
> > > > I'm going to jump straight to the end. I think that one of the
> problems
> > > of
> > > > doing an abstraction similar to remoteGraph is that in reality the
> > > overhead
> > > > of communicating with the server is too big to make this viable. It's
> > > > imperative that you have the ability to group multiline scripts into
> a
> > > > single query or the reactivity of your applications will greatly
> > suffer.
> > > >
> > > > The Gremlin-java base also has a few limitations that some query
> > builders
> > > > try to fix (which can only be done if you abandon the idea of a
> natural
> > > > gremlin language variant in favor of a query builder). Namely having
> > the
> > > > ability to edit/modify/add steps to an already defined traversal.
> > Though
> > > in
> > > > time it might be nice to have these be part of the original
> > > implementation.
> > > >
> > > > I personally love the idea of gremlin language variants. I just don't
> > > think
> > > > they're production value is any good without some extended
> > functionality
> > > > (beyond what gremlin-java currently is).
> > > >
> > > > On Wed, Apr 20, 2016 at 3:23 PM, Stephen Mallette <
> > spmallette@gmail.com>
> > > > wrote:
> > > >
> > > > > This thread on Gremlin Language Variants has been very interesting:
> > > > >
> > > > > https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
> > > > >
> > > > > I think that this work goes a long way to address two issues I've
> > been
> > > > > concerned about:
> > > > >
> > > > > 1. Greater consistency in how different languages do Gremlin
> > > > > 2. Less fragmentation in terms of libraries and how they work so
> that
> > > > users
> > > > > aren't confused with how to get started (though I don't think the
> > goal
> > > > here
> > > > > is to restrict choices or slow down innovation)
> > > > >
> > > > > One of the first things we should probably do is start thinking in
> > > terms
> > > > of
> > > > > the types of libraries that are built on TinkerPop (outside of
> those
> > > > things
> > > > > that are Graph Systems) and those are listed here currently:
> > > > >
> > > > > http://tinkerpop.apache.org/#graph-libraries
> > > > >
> > > > > Marko mentioned to me that he saw the libraries we listed here
> > breaking
> > > > > into three categories:
> > > > >
> > > > > 1. Gremlin Language Variants - which the other thread demonstrates
> > > quite
> > > > > nicely
> > > > > 2. Gremlin Drivers - the Gremlin Server protocol implementations -
> > > those
> > > > > things that send traversals to Gremlin Server and get back results.
> > > > > 3. OGM and others - I say "others" because there might be plugins
> and
> > > > other
> > > > > similar odds and ends
> > > > >
> > > > > I like Marko's category system here and I think that having these
> > kinds
> > > > of
> > > > > categories will help folks organize their libraries to fit into one
> > of
> > > > > these spaces and make it easier for users to know what they need to
> > get
> > > > in
> > > > > order to start doing TinkerPop in their language.
> > > > >
> > > > > Anyway, the category thing is just setting the stage for this big
> > > > > bombshell.  I think TinkerPop should consider maintaining the
> Gremlin
> > > > > Language Variants.
> > > > >
> > > > > Heresy! right?
> > > > >
> > > > > Well, I think it's the best way to achieve consistency across
> > > languages.
> > > > > Under this model, TinkerPop provides the base language variant and
> > > people
> > > > > can choose to extend upon it, but the base stays tied to our
> > archetype
> > > of
> > > > > Java and we end up with a much more clear story for virtually any
> > > > > programming language.
> > > > >
> > > > > So how do we do this? Slowly and deliberately. We should look to
> only
> > > > > include language variants where we:
> > > > >
> > > > > + have good automation in place (like what Marko did for Python),
> > > > > + some competence on the committer list in that language
> > > > > + a nice testing framework that operates in our standard
> > build/release
> > > > > process.
> > > > >
> > > > > That's setting a high bar, but if we don't keep it high, I think we
> > > will
> > > > be
> > > > > left unable to properly support and maintain what we hang out
> there.
> > > > >
> > > > > I'd also like to express that we should not be looking to maintain
> > > > language
> > > > > drivers. I think that should remain a third-party community effort
> > just
> > > > > like Graph Systems. In other words, we remain a repository for
> > > reference
> > > > > implementations for everything else. Why? Because, as it sits right
> > > now,
> > > > > just based on the level of effort for what Marko did with Python,
> > > > > maintaining a "base" Gremlin Language Variants shouldn't be hard.
> We
> > > > won't
> > > > > be building tons of add-on capabilities to the base variants - they
> > > will
> > > > > pretty much just stay on par with the java archetype.  Drivers on
> the
> > > > other
> > > > > hand have lots of implementation details, with many different
> > > > technologies
> > > > > that could be used, etc.  They have similar complexity to Graph
> > System
> > > > > implementations in many ways. I also think that the drivers can
> > afford
> > > to
> > > > > have different APIs and approaches without being detrimental to the
> > > > > community.
> > > > >
> > > > > If gremlin-js-driver wants to do:
> > > > >
> > > > > client.submit("g.V()")
> > > > >
> > > > > and gremlin-python-driver wants to do:
> > > > >
> > > > > client.send("g.V()")
> > > > >
> > > > > that's not a big deal.
> > > > >
> > > > > The last point that I'll make is that I think Gremlin Language
> > > Variants,
> > > > > that don't operate on the JVM (e.g. Jython) and use Gremlin Server,
> > > > should
> > > > > have some abstraction that is similar to RemoteGraph.  RemoteGraph
> > > > exposes
> > > > > a DriverConnection interface that is currently implemented by
> > > > > gremlin-driver.  The DriverConnection is responsible for sending a
> > > > > traversal to the server and returning results. It would be nice if
> > the
> > > > > language variants had a similar interface that the various
> community
> > > > > drivers could implement. In that way, the user never has to do any
> > form
> > > > of:
> > > > >
> > > > > client.submit(someGremlinString)
> > > > >
> > > > > in any language. We really need to try to make that pattern go away
> > > > across
> > > > > the TinkerPop community.
> > > > >
> > > > > So - that's was a long email. Looking forward to hearing some
> > > discussion
> > > > on
> > > > > this.
> > > > >
> > > > > Stephen
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Stephen Mallette <sp...@gmail.com>]

So "query building" is where you hold string representations of steps or
groups of steps and then have some logic that concatenates them together. I
guess my question is why are "strings" a requirement for that? It seems
like you could do the same query builder stuff with an actual Traversal
object in whatever Gremlin Language Variant you were working with. How is
string concatenation different than this:

gremlin> graph = TinkerFactory.createModern()
==>tinkergraph[vertices:6 edges:6]
gremlin> g = graph.traversal()
==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
gremlin> base = g.V().hasLabel('person');[]
gremlin> addFilters = { age, name, t ->
gremlin>   t = age > 0 ? t.has('age',age) : t
gremlin>   !name.isEmpty() ? t.has('name',name) : t
gremlin> }
==>groovysh_evaluate$_run_closure1@503d56b5
gremlin> traversal = addFilters(29,'',base);[]
gremlin> traversal.toString()
==>[GraphStep(vertex,[]), HasStep([~label.eq(person)]),
HasStep([age.eq(29)])]
gremlin> traversal
==>v[1]

What about DSLs? Maybe filtering logic drops behind a custom step specific
to the domain:

g.V().personsWith(29, '')

which would basically compile to the same thing as the gremlin output above:

[GraphStep(vertex,[]), HasStep([~label.eq(person)]), HasStep([age.eq(29)])]

Is there some more complex aspect "query building" that can't be handled
this way (I know I took a fairly simple example)?



On Wed, Apr 20, 2016 at 7:54 PM, Dylan Millikin <dy...@gmail.com>
wrote:

> I've approached this a few times in the past though never really in depth.
> The idea is that you want separation of logic in your queries, mostly for
> maintenance, testing and overall convenience. This takes the form of
> partial/incomplete traversals that have no start nor end.
> One of the simplest applications would be to have a base query and apply
> filters to this query. For example we can imagine a recap page of
> everyone's companies:
>
> base query : g.V().has(label, "person").out('company')
>
> To this you have a set of filters that are partial traversals and allow
> users to better refine their search:
>
> older than 30 filter: has("age", gt(30))
> male filter: has("gender", "male")
>
> Now depending on user input you'll want to apply either or both of the
> above to "person" in order to obtain something along the lines of:
>
> g.V().has(label, "person").has("age", gt(30)).has("gender",
> "male").out('company')
>
> Of course there are plenty of ways of doing the above depending on
> requirements and complexity. Such as where() or as("a")....select("a")
> etc..
>
> These are simple examples where steps are appended in various places but
> you can imagine the same with traversal manipulation such as turning
> out('company') into out('company', 'organization') where the step gets
> altered. This would work the same way if you used union(out('company'))
> instead. You would have to alter the union step like this :
> union(out('company'),
> out('organization'))
>
> All of the above are relatively simple but picture having a lot of very
> complex traversal "filters" that sometimes have dependencies on other
> filters.
> All this is doable but it currently requires looking at the gremlin
> building process as a query building one rather than native support for the
> gremlin language.
>
> Does that make any sense?
>
> On Wed, Apr 20, 2016 at 7:15 PM, Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > >  It's imperative that you have the ability to group multiline scripts
> > into a single query or the reactivity of your applications will greatly
> > suffer.
> >
> > A fair point and something we might yet address as part of all this
> > thinking. RemoteGraph is pretty new and it demonstrated a critical aspect
> > of communications with Gremlin Server. Now we need to think about how to
> > improve upon it.
> >
> > > Namely having the ability to edit/modify/add steps to an already
> defined
> > traversal.
> >
> > I don't quite follow that point...could you please elaborate?
> >
> >
> >
> >
> > On Wed, Apr 20, 2016 at 4:15 PM, Dylan Millikin <
> dylan.millikin@gmail.com>
> > wrote:
> >
> > > Nice post,
> > >
> > > I'm going to jump straight to the end. I think that one of the problems
> > of
> > > doing an abstraction similar to remoteGraph is that in reality the
> > overhead
> > > of communicating with the server is too big to make this viable. It's
> > > imperative that you have the ability to group multiline scripts into a
> > > single query or the reactivity of your applications will greatly
> suffer.
> > >
> > > The Gremlin-java base also has a few limitations that some query
> builders
> > > try to fix (which can only be done if you abandon the idea of a natural
> > > gremlin language variant in favor of a query builder). Namely having
> the
> > > ability to edit/modify/add steps to an already defined traversal.
> Though
> > in
> > > time it might be nice to have these be part of the original
> > implementation.
> > >
> > > I personally love the idea of gremlin language variants. I just don't
> > think
> > > they're production value is any good without some extended
> functionality
> > > (beyond what gremlin-java currently is).
> > >
> > > On Wed, Apr 20, 2016 at 3:23 PM, Stephen Mallette <
> spmallette@gmail.com>
> > > wrote:
> > >
> > > > This thread on Gremlin Language Variants has been very interesting:
> > > >
> > > > https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
> > > >
> > > > I think that this work goes a long way to address two issues I've
> been
> > > > concerned about:
> > > >
> > > > 1. Greater consistency in how different languages do Gremlin
> > > > 2. Less fragmentation in terms of libraries and how they work so that
> > > users
> > > > aren't confused with how to get started (though I don't think the
> goal
> > > here
> > > > is to restrict choices or slow down innovation)
> > > >
> > > > One of the first things we should probably do is start thinking in
> > terms
> > > of
> > > > the types of libraries that are built on TinkerPop (outside of those
> > > things
> > > > that are Graph Systems) and those are listed here currently:
> > > >
> > > > http://tinkerpop.apache.org/#graph-libraries
> > > >
> > > > Marko mentioned to me that he saw the libraries we listed here
> breaking
> > > > into three categories:
> > > >
> > > > 1. Gremlin Language Variants - which the other thread demonstrates
> > quite
> > > > nicely
> > > > 2. Gremlin Drivers - the Gremlin Server protocol implementations -
> > those
> > > > things that send traversals to Gremlin Server and get back results.
> > > > 3. OGM and others - I say "others" because there might be plugins and
> > > other
> > > > similar odds and ends
> > > >
> > > > I like Marko's category system here and I think that having these
> kinds
> > > of
> > > > categories will help folks organize their libraries to fit into one
> of
> > > > these spaces and make it easier for users to know what they need to
> get
> > > in
> > > > order to start doing TinkerPop in their language.
> > > >
> > > > Anyway, the category thing is just setting the stage for this big
> > > > bombshell.  I think TinkerPop should consider maintaining the Gremlin
> > > > Language Variants.
> > > >
> > > > Heresy! right?
> > > >
> > > > Well, I think it's the best way to achieve consistency across
> > languages.
> > > > Under this model, TinkerPop provides the base language variant and
> > people
> > > > can choose to extend upon it, but the base stays tied to our
> archetype
> > of
> > > > Java and we end up with a much more clear story for virtually any
> > > > programming language.
> > > >
> > > > So how do we do this? Slowly and deliberately. We should look to only
> > > > include language variants where we:
> > > >
> > > > + have good automation in place (like what Marko did for Python),
> > > > + some competence on the committer list in that language
> > > > + a nice testing framework that operates in our standard
> build/release
> > > > process.
> > > >
> > > > That's setting a high bar, but if we don't keep it high, I think we
> > will
> > > be
> > > > left unable to properly support and maintain what we hang out there.
> > > >
> > > > I'd also like to express that we should not be looking to maintain
> > > language
> > > > drivers. I think that should remain a third-party community effort
> just
> > > > like Graph Systems. In other words, we remain a repository for
> > reference
> > > > implementations for everything else. Why? Because, as it sits right
> > now,
> > > > just based on the level of effort for what Marko did with Python,
> > > > maintaining a "base" Gremlin Language Variants shouldn't be hard. We
> > > won't
> > > > be building tons of add-on capabilities to the base variants - they
> > will
> > > > pretty much just stay on par with the java archetype.  Drivers on the
> > > other
> > > > hand have lots of implementation details, with many different
> > > technologies
> > > > that could be used, etc.  They have similar complexity to Graph
> System
> > > > implementations in many ways. I also think that the drivers can
> afford
> > to
> > > > have different APIs and approaches without being detrimental to the
> > > > community.
> > > >
> > > > If gremlin-js-driver wants to do:
> > > >
> > > > client.submit("g.V()")
> > > >
> > > > and gremlin-python-driver wants to do:
> > > >
> > > > client.send("g.V()")
> > > >
> > > > that's not a big deal.
> > > >
> > > > The last point that I'll make is that I think Gremlin Language
> > Variants,
> > > > that don't operate on the JVM (e.g. Jython) and use Gremlin Server,
> > > should
> > > > have some abstraction that is similar to RemoteGraph.  RemoteGraph
> > > exposes
> > > > a DriverConnection interface that is currently implemented by
> > > > gremlin-driver.  The DriverConnection is responsible for sending a
> > > > traversal to the server and returning results. It would be nice if
> the
> > > > language variants had a similar interface that the various community
> > > > drivers could implement. In that way, the user never has to do any
> form
> > > of:
> > > >
> > > > client.submit(someGremlinString)
> > > >
> > > > in any language. We really need to try to make that pattern go away
> > > across
> > > > the TinkerPop community.
> > > >
> > > > So - that's was a long email. Looking forward to hearing some
> > discussion
> > > on
> > > > this.
> > > >
> > > > Stephen
> > > >
> > >
> >
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Dylan Millikin <dy...@gmail.com>]

I've approached this a few times in the past though never really in depth.
The idea is that you want separation of logic in your queries, mostly for
maintenance, testing and overall convenience. This takes the form of
partial/incomplete traversals that have no start nor end.
One of the simplest applications would be to have a base query and apply
filters to this query. For example we can imagine a recap page of
everyone's companies:

base query : g.V().has(label, "person").out('company')

To this you have a set of filters that are partial traversals and allow
users to better refine their search:

older than 30 filter: has("age", gt(30))
male filter: has("gender", "male")

Now depending on user input you'll want to apply either or both of the
above to "person" in order to obtain something along the lines of:

g.V().has(label, "person").has("age", gt(30)).has("gender",
"male").out('company')

Of course there are plenty of ways of doing the above depending on
requirements and complexity. Such as where() or as("a")....select("a") etc..

These are simple examples where steps are appended in various places but
you can imagine the same with traversal manipulation such as turning
out('company') into out('company', 'organization') where the step gets
altered. This would work the same way if you used union(out('company'))
instead. You would have to alter the union step like this :
union(out('company'),
out('organization'))

All of the above are relatively simple but picture having a lot of very
complex traversal "filters" that sometimes have dependencies on other
filters.
All this is doable but it currently requires looking at the gremlin
building process as a query building one rather than native support for the
gremlin language.

Does that make any sense?

On Wed, Apr 20, 2016 at 7:15 PM, Stephen Mallette <sp...@gmail.com>
wrote:

> >  It's imperative that you have the ability to group multiline scripts
> into a single query or the reactivity of your applications will greatly
> suffer.
>
> A fair point and something we might yet address as part of all this
> thinking. RemoteGraph is pretty new and it demonstrated a critical aspect
> of communications with Gremlin Server. Now we need to think about how to
> improve upon it.
>
> > Namely having the ability to edit/modify/add steps to an already defined
> traversal.
>
> I don't quite follow that point...could you please elaborate?
>
>
>
>
> On Wed, Apr 20, 2016 at 4:15 PM, Dylan Millikin <dy...@gmail.com>
> wrote:
>
> > Nice post,
> >
> > I'm going to jump straight to the end. I think that one of the problems
> of
> > doing an abstraction similar to remoteGraph is that in reality the
> overhead
> > of communicating with the server is too big to make this viable. It's
> > imperative that you have the ability to group multiline scripts into a
> > single query or the reactivity of your applications will greatly suffer.
> >
> > The Gremlin-java base also has a few limitations that some query builders
> > try to fix (which can only be done if you abandon the idea of a natural
> > gremlin language variant in favor of a query builder). Namely having the
> > ability to edit/modify/add steps to an already defined traversal. Though
> in
> > time it might be nice to have these be part of the original
> implementation.
> >
> > I personally love the idea of gremlin language variants. I just don't
> think
> > they're production value is any good without some extended functionality
> > (beyond what gremlin-java currently is).
> >
> > On Wed, Apr 20, 2016 at 3:23 PM, Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > This thread on Gremlin Language Variants has been very interesting:
> > >
> > > https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
> > >
> > > I think that this work goes a long way to address two issues I've been
> > > concerned about:
> > >
> > > 1. Greater consistency in how different languages do Gremlin
> > > 2. Less fragmentation in terms of libraries and how they work so that
> > users
> > > aren't confused with how to get started (though I don't think the goal
> > here
> > > is to restrict choices or slow down innovation)
> > >
> > > One of the first things we should probably do is start thinking in
> terms
> > of
> > > the types of libraries that are built on TinkerPop (outside of those
> > things
> > > that are Graph Systems) and those are listed here currently:
> > >
> > > http://tinkerpop.apache.org/#graph-libraries
> > >
> > > Marko mentioned to me that he saw the libraries we listed here breaking
> > > into three categories:
> > >
> > > 1. Gremlin Language Variants - which the other thread demonstrates
> quite
> > > nicely
> > > 2. Gremlin Drivers - the Gremlin Server protocol implementations -
> those
> > > things that send traversals to Gremlin Server and get back results.
> > > 3. OGM and others - I say "others" because there might be plugins and
> > other
> > > similar odds and ends
> > >
> > > I like Marko's category system here and I think that having these kinds
> > of
> > > categories will help folks organize their libraries to fit into one of
> > > these spaces and make it easier for users to know what they need to get
> > in
> > > order to start doing TinkerPop in their language.
> > >
> > > Anyway, the category thing is just setting the stage for this big
> > > bombshell.  I think TinkerPop should consider maintaining the Gremlin
> > > Language Variants.
> > >
> > > Heresy! right?
> > >
> > > Well, I think it's the best way to achieve consistency across
> languages.
> > > Under this model, TinkerPop provides the base language variant and
> people
> > > can choose to extend upon it, but the base stays tied to our archetype
> of
> > > Java and we end up with a much more clear story for virtually any
> > > programming language.
> > >
> > > So how do we do this? Slowly and deliberately. We should look to only
> > > include language variants where we:
> > >
> > > + have good automation in place (like what Marko did for Python),
> > > + some competence on the committer list in that language
> > > + a nice testing framework that operates in our standard build/release
> > > process.
> > >
> > > That's setting a high bar, but if we don't keep it high, I think we
> will
> > be
> > > left unable to properly support and maintain what we hang out there.
> > >
> > > I'd also like to express that we should not be looking to maintain
> > language
> > > drivers. I think that should remain a third-party community effort just
> > > like Graph Systems. In other words, we remain a repository for
> reference
> > > implementations for everything else. Why? Because, as it sits right
> now,
> > > just based on the level of effort for what Marko did with Python,
> > > maintaining a "base" Gremlin Language Variants shouldn't be hard. We
> > won't
> > > be building tons of add-on capabilities to the base variants - they
> will
> > > pretty much just stay on par with the java archetype.  Drivers on the
> > other
> > > hand have lots of implementation details, with many different
> > technologies
> > > that could be used, etc.  They have similar complexity to Graph System
> > > implementations in many ways. I also think that the drivers can afford
> to
> > > have different APIs and approaches without being detrimental to the
> > > community.
> > >
> > > If gremlin-js-driver wants to do:
> > >
> > > client.submit("g.V()")
> > >
> > > and gremlin-python-driver wants to do:
> > >
> > > client.send("g.V()")
> > >
> > > that's not a big deal.
> > >
> > > The last point that I'll make is that I think Gremlin Language
> Variants,
> > > that don't operate on the JVM (e.g. Jython) and use Gremlin Server,
> > should
> > > have some abstraction that is similar to RemoteGraph.  RemoteGraph
> > exposes
> > > a DriverConnection interface that is currently implemented by
> > > gremlin-driver.  The DriverConnection is responsible for sending a
> > > traversal to the server and returning results. It would be nice if the
> > > language variants had a similar interface that the various community
> > > drivers could implement. In that way, the user never has to do any form
> > of:
> > >
> > > client.submit(someGremlinString)
> > >
> > > in any language. We really need to try to make that pattern go away
> > across
> > > the TinkerPop community.
> > >
> > > So - that's was a long email. Looking forward to hearing some
> discussion
> > on
> > > this.
> > >
> > > Stephen
> > >
> >
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Stephen Mallette <sp...@gmail.com>]

>  It's imperative that you have the ability to group multiline scripts
into a single query or the reactivity of your applications will greatly
suffer.

A fair point and something we might yet address as part of all this
thinking. RemoteGraph is pretty new and it demonstrated a critical aspect
of communications with Gremlin Server. Now we need to think about how to
improve upon it.

> Namely having the ability to edit/modify/add steps to an already defined
traversal.

I don't quite follow that point...could you please elaborate?




On Wed, Apr 20, 2016 at 4:15 PM, Dylan Millikin <dy...@gmail.com>
wrote:

> Nice post,
>
> I'm going to jump straight to the end. I think that one of the problems of
> doing an abstraction similar to remoteGraph is that in reality the overhead
> of communicating with the server is too big to make this viable. It's
> imperative that you have the ability to group multiline scripts into a
> single query or the reactivity of your applications will greatly suffer.
>
> The Gremlin-java base also has a few limitations that some query builders
> try to fix (which can only be done if you abandon the idea of a natural
> gremlin language variant in favor of a query builder). Namely having the
> ability to edit/modify/add steps to an already defined traversal. Though in
> time it might be nice to have these be part of the original implementation.
>
> I personally love the idea of gremlin language variants. I just don't think
> they're production value is any good without some extended functionality
> (beyond what gremlin-java currently is).
>
> On Wed, Apr 20, 2016 at 3:23 PM, Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > This thread on Gremlin Language Variants has been very interesting:
> >
> > https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
> >
> > I think that this work goes a long way to address two issues I've been
> > concerned about:
> >
> > 1. Greater consistency in how different languages do Gremlin
> > 2. Less fragmentation in terms of libraries and how they work so that
> users
> > aren't confused with how to get started (though I don't think the goal
> here
> > is to restrict choices or slow down innovation)
> >
> > One of the first things we should probably do is start thinking in terms
> of
> > the types of libraries that are built on TinkerPop (outside of those
> things
> > that are Graph Systems) and those are listed here currently:
> >
> > http://tinkerpop.apache.org/#graph-libraries
> >
> > Marko mentioned to me that he saw the libraries we listed here breaking
> > into three categories:
> >
> > 1. Gremlin Language Variants - which the other thread demonstrates quite
> > nicely
> > 2. Gremlin Drivers - the Gremlin Server protocol implementations - those
> > things that send traversals to Gremlin Server and get back results.
> > 3. OGM and others - I say "others" because there might be plugins and
> other
> > similar odds and ends
> >
> > I like Marko's category system here and I think that having these kinds
> of
> > categories will help folks organize their libraries to fit into one of
> > these spaces and make it easier for users to know what they need to get
> in
> > order to start doing TinkerPop in their language.
> >
> > Anyway, the category thing is just setting the stage for this big
> > bombshell.  I think TinkerPop should consider maintaining the Gremlin
> > Language Variants.
> >
> > Heresy! right?
> >
> > Well, I think it's the best way to achieve consistency across languages.
> > Under this model, TinkerPop provides the base language variant and people
> > can choose to extend upon it, but the base stays tied to our archetype of
> > Java and we end up with a much more clear story for virtually any
> > programming language.
> >
> > So how do we do this? Slowly and deliberately. We should look to only
> > include language variants where we:
> >
> > + have good automation in place (like what Marko did for Python),
> > + some competence on the committer list in that language
> > + a nice testing framework that operates in our standard build/release
> > process.
> >
> > That's setting a high bar, but if we don't keep it high, I think we will
> be
> > left unable to properly support and maintain what we hang out there.
> >
> > I'd also like to express that we should not be looking to maintain
> language
> > drivers. I think that should remain a third-party community effort just
> > like Graph Systems. In other words, we remain a repository for reference
> > implementations for everything else. Why? Because, as it sits right now,
> > just based on the level of effort for what Marko did with Python,
> > maintaining a "base" Gremlin Language Variants shouldn't be hard. We
> won't
> > be building tons of add-on capabilities to the base variants - they will
> > pretty much just stay on par with the java archetype.  Drivers on the
> other
> > hand have lots of implementation details, with many different
> technologies
> > that could be used, etc.  They have similar complexity to Graph System
> > implementations in many ways. I also think that the drivers can afford to
> > have different APIs and approaches without being detrimental to the
> > community.
> >
> > If gremlin-js-driver wants to do:
> >
> > client.submit("g.V()")
> >
> > and gremlin-python-driver wants to do:
> >
> > client.send("g.V()")
> >
> > that's not a big deal.
> >
> > The last point that I'll make is that I think Gremlin Language Variants,
> > that don't operate on the JVM (e.g. Jython) and use Gremlin Server,
> should
> > have some abstraction that is similar to RemoteGraph.  RemoteGraph
> exposes
> > a DriverConnection interface that is currently implemented by
> > gremlin-driver.  The DriverConnection is responsible for sending a
> > traversal to the server and returning results. It would be nice if the
> > language variants had a similar interface that the various community
> > drivers could implement. In that way, the user never has to do any form
> of:
> >
> > client.submit(someGremlinString)
> >
> > in any language. We really need to try to make that pattern go away
> across
> > the TinkerPop community.
> >
> > So - that's was a long email. Looking forward to hearing some discussion
> on
> > this.
> >
> > Stephen
> >
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Dylan Millikin <dy...@gmail.com>]

Nice post,

I'm going to jump straight to the end. I think that one of the problems of
doing an abstraction similar to remoteGraph is that in reality the overhead
of communicating with the server is too big to make this viable. It's
imperative that you have the ability to group multiline scripts into a
single query or the reactivity of your applications will greatly suffer.

The Gremlin-java base also has a few limitations that some query builders
try to fix (which can only be done if you abandon the idea of a natural
gremlin language variant in favor of a query builder). Namely having the
ability to edit/modify/add steps to an already defined traversal. Though in
time it might be nice to have these be part of the original implementation.

I personally love the idea of gremlin language variants. I just don't think
they're production value is any good without some extended functionality
(beyond what gremlin-java currently is).

On Wed, Apr 20, 2016 at 3:23 PM, Stephen Mallette <sp...@gmail.com>
wrote:

> This thread on Gremlin Language Variants has been very interesting:
>
> https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
>
> I think that this work goes a long way to address two issues I've been
> concerned about:
>
> 1. Greater consistency in how different languages do Gremlin
> 2. Less fragmentation in terms of libraries and how they work so that users
> aren't confused with how to get started (though I don't think the goal here
> is to restrict choices or slow down innovation)
>
> One of the first things we should probably do is start thinking in terms of
> the types of libraries that are built on TinkerPop (outside of those things
> that are Graph Systems) and those are listed here currently:
>
> http://tinkerpop.apache.org/#graph-libraries
>
> Marko mentioned to me that he saw the libraries we listed here breaking
> into three categories:
>
> 1. Gremlin Language Variants - which the other thread demonstrates quite
> nicely
> 2. Gremlin Drivers - the Gremlin Server protocol implementations - those
> things that send traversals to Gremlin Server and get back results.
> 3. OGM and others - I say "others" because there might be plugins and other
> similar odds and ends
>
> I like Marko's category system here and I think that having these kinds of
> categories will help folks organize their libraries to fit into one of
> these spaces and make it easier for users to know what they need to get in
> order to start doing TinkerPop in their language.
>
> Anyway, the category thing is just setting the stage for this big
> bombshell.  I think TinkerPop should consider maintaining the Gremlin
> Language Variants.
>
> Heresy! right?
>
> Well, I think it's the best way to achieve consistency across languages.
> Under this model, TinkerPop provides the base language variant and people
> can choose to extend upon it, but the base stays tied to our archetype of
> Java and we end up with a much more clear story for virtually any
> programming language.
>
> So how do we do this? Slowly and deliberately. We should look to only
> include language variants where we:
>
> + have good automation in place (like what Marko did for Python),
> + some competence on the committer list in that language
> + a nice testing framework that operates in our standard build/release
> process.
>
> That's setting a high bar, but if we don't keep it high, I think we will be
> left unable to properly support and maintain what we hang out there.
>
> I'd also like to express that we should not be looking to maintain language
> drivers. I think that should remain a third-party community effort just
> like Graph Systems. In other words, we remain a repository for reference
> implementations for everything else. Why? Because, as it sits right now,
> just based on the level of effort for what Marko did with Python,
> maintaining a "base" Gremlin Language Variants shouldn't be hard. We won't
> be building tons of add-on capabilities to the base variants - they will
> pretty much just stay on par with the java archetype.  Drivers on the other
> hand have lots of implementation details, with many different technologies
> that could be used, etc.  They have similar complexity to Graph System
> implementations in many ways. I also think that the drivers can afford to
> have different APIs and approaches without being detrimental to the
> community.
>
> If gremlin-js-driver wants to do:
>
> client.submit("g.V()")
>
> and gremlin-python-driver wants to do:
>
> client.send("g.V()")
>
> that's not a big deal.
>
> The last point that I'll make is that I think Gremlin Language Variants,
> that don't operate on the JVM (e.g. Jython) and use Gremlin Server, should
> have some abstraction that is similar to RemoteGraph.  RemoteGraph exposes
> a DriverConnection interface that is currently implemented by
> gremlin-driver.  The DriverConnection is responsible for sending a
> traversal to the server and returning results. It would be nice if the
> language variants had a similar interface that the various community
> drivers could implement. In that way, the user never has to do any form of:
>
> client.submit(someGremlinString)
>
> in any language. We really need to try to make that pattern go away across
> the TinkerPop community.
>
> So - that's was a long email. Looking forward to hearing some discussion on
> this.
>
> Stephen
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Marko Rodriguez <ok...@gmail.com>]

Hi,

I added the following ticket for this DISCUSS.
	https://issues.apache.org/jira/browse/TINKERPOP-1278

Bye,
Marko.

http://markorodriguez.com

On Apr 23, 2016, at 9:23 AM, Stephen Mallette <sp...@gmail.com> wrote:

> I would think that it's important to develop a pattern for these projects.
> Since Marko is reasonably far down the path of python, I assume that we
> would build the model project from that.
> 
> On Sat, Apr 23, 2016 at 10:26 AM, Jean-Baptiste Musso <jb...@gmail.com>
> wrote:
> 
>> On Wed, Apr 20, 2016 at 9:23 PM, Stephen Mallette <sp...@gmail.com>
>> wrote:
>>> This thread on Gremlin Language Variants has been very interesting:
>>> 
>>> https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
>>> 
>>> Anyway, the category thing is just setting the stage for this big
>>> bombshell.  I think TinkerPop should consider maintaining the Gremlin
>>> Language Variants.
>>> 
>>> Heresy! right?
>> 
>> Thanks for writing this write-up.
>> I really like this idea, I think it's fine to maintain language
>> variants while leaving drivers to third party. If we agree on this,
>> I'll see if I can come up with something clean in JavaScript.
>> 
>> Jean-Baptiste
>> 

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Stephen Mallette <sp...@gmail.com>]

I would think that it's important to develop a pattern for these projects.
Since Marko is reasonably far down the path of python, I assume that we
would build the model project from that.

On Sat, Apr 23, 2016 at 10:26 AM, Jean-Baptiste Musso <jb...@gmail.com>
wrote:

> On Wed, Apr 20, 2016 at 9:23 PM, Stephen Mallette <sp...@gmail.com>
> wrote:
> > This thread on Gremlin Language Variants has been very interesting:
> >
> > https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
> >
> > Anyway, the category thing is just setting the stage for this big
> > bombshell.  I think TinkerPop should consider maintaining the Gremlin
> > Language Variants.
> >
> > Heresy! right?
>
> Thanks for writing this write-up.
> I really like this idea, I think it's fine to maintain language
> variants while leaving drivers to third party. If we agree on this,
> I'll see if I can come up with something clean in JavaScript.
>
> Jean-Baptiste
>

Re: [DISCUSS] TinkerPop mantained Gremlin Language Variants

[Jean-Baptiste Musso <jb...@gmail.com>]

On Wed, Apr 20, 2016 at 9:23 PM, Stephen Mallette <sp...@gmail.com> wrote:
> This thread on Gremlin Language Variants has been very interesting:
>
> https://pony-poc.apache.org/thread.html/Zcazrw7k442xcwc
>
> Anyway, the category thing is just setting the stage for this big
> bombshell.  I think TinkerPop should consider maintaining the Gremlin
> Language Variants.
>
> Heresy! right?

Thanks for writing this write-up.
I really like this idea, I think it's fine to maintain language
variants while leaving drivers to third party. If we agree on this,
I'll see if I can come up with something clean in JavaScript.

Jean-Baptiste