You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by Stephen Mallette <sp...@gmail.com> on 2020/12/24 19:41:24 UTC
[DISCUSS] ANTLR and gremlin-script
As a project, over the years, we've often been asked the question as to why
Gremlin doesn't have an ANTLR style grammar. There have been varying
answers over the years to explain the reasoning but in recent years I've
started to see where our dependence on Java for driving Gremlin design has
not translated well as we have expanded Gremlin into other programming
ecosystems. Using Java has often allowed idioms of that language to leak
into Gremlin itself which introduces friction when implemented outside of
the JVM. I think that there is some advantage to designing Gremlin more
with just graphs/usage in mind and then determining how that design choice
looks in each programming language.
I think that using an ANTLR grammar to drive that design work for Gremlin
makes a lot of sense in this context. We would effectively have something
like a gremlin-script which would become the new language archetype. New
steps, language changes, etc. would be discussed in its context and then
implemented in the grammar and later in each programming language we
support in the style a developer would expect. An interesting upside of
this approach is that we can implement gremlin-script in the ScriptEngine
and replace GremlinGroovyScriptEngine which would help us strengthen our
security story in Gremlin Server. Groovy processing would just be a
fallback to Gremlin scripts that could not be processed by the AST. In fact
users who didn't need Groovy could simply not install it at all and thus
boast a much more secure system.
I think that inclusion of a grammar in our project is an exciting new
direction for us to take and will help in a variety of areas beyond those
I've already related.
If we like this direction, Amazon Neptune already maintains such a grammar
and would be willing to contribute it to the project to live in open
source. The contribution would go through the same IP Clearance process
gremlint is going through since it was developed outside of TinkerPop. I'd
be happy to guide that process through if we draw to consensus here.
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Josh Perryman <jo...@gmail.com>.
First of all: I love this direction, for all of the reasons cited and the
innovation I'm hoping that it opens up.
Also, that Amazon Neptune has already done the work and would be willing to
share it with the rest of the world is a remarkable gift.
I'm just a user / admirer of the technology, but as an opinionated graph
junkie, this is a really great thing.
-Josh
On Thu, Dec 24, 2020 at 1:41 PM Stephen Mallette <sp...@gmail.com>
wrote:
> As a project, over the years, we've often been asked the question as to why
> Gremlin doesn't have an ANTLR style grammar. There have been varying
> answers over the years to explain the reasoning but in recent years I've
> started to see where our dependence on Java for driving Gremlin design has
> not translated well as we have expanded Gremlin into other programming
> ecosystems. Using Java has often allowed idioms of that language to leak
> into Gremlin itself which introduces friction when implemented outside of
> the JVM. I think that there is some advantage to designing Gremlin more
> with just graphs/usage in mind and then determining how that design choice
> looks in each programming language.
>
> I think that using an ANTLR grammar to drive that design work for Gremlin
> makes a lot of sense in this context. We would effectively have something
> like a gremlin-script which would become the new language archetype. New
> steps, language changes, etc. would be discussed in its context and then
> implemented in the grammar and later in each programming language we
> support in the style a developer would expect. An interesting upside of
> this approach is that we can implement gremlin-script in the ScriptEngine
> and replace GremlinGroovyScriptEngine which would help us strengthen our
> security story in Gremlin Server. Groovy processing would just be a
> fallback to Gremlin scripts that could not be processed by the AST. In fact
> users who didn't need Groovy could simply not install it at all and thus
> boast a much more secure system.
>
> I think that inclusion of a grammar in our project is an exciting new
> direction for us to take and will help in a variety of areas beyond those
> I've already related.
>
> If we like this direction, Amazon Neptune already maintains such a grammar
> and would be willing to contribute it to the project to live in open
> source. The contribution would go through the same IP Clearance process
> gremlint is going through since it was developed outside of TinkerPop. I'd
> be happy to guide that process through if we draw to consensus here.
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Stephen Mallette <sp...@gmail.com>.
Some thoughts inline:
On Mon, Mar 22, 2021 at 2:37 PM pieter gmail <pi...@gmail.com>
wrote:
> Hi,
>
> Exciting as this is I am not quite sure what it means.
>
> Naively perhaps it the idea,
> Arbitary gremlin string -> antlr parser -> some AST walker -> gremlin
> byte code -> java in memory steps ... -> voila
>
> Is the grammar going to be the primary and only
> interface/specification, or will the native java implementation bypass
> the grammar going straight to the steps instead?
>
Perhaps the most practical point of the grammar is that I don't have much
faith in Groovy these days. It's been a security problem for all the major
graph providers, there are many issues with performance (some of which
we've managed to handle, some we've handled a bit less well and some
perhaps not at all), and nowadays we aren't getting much response from
their community on major issues that impact us:
https://issues.apache.org/jira/browse/TINKERPOP-2526
So, the grammar seems like a good way to find a bit of independence there
as we can process Gremlin strings (most of them anyway at this point)
without Groovy. I used to think that Gremlin bytecode would allow us to get
rid of Gremlin strings, but that seems like an impossibility. There are
just too many users with that model. The same could be said of bytecode at
this point. So I sense, neither is going away any time soon, so we may as
well accept it and try to make both environments work as well as possible.
Perhaps in TP4 someday we could make the decision of grammar or bytecode,
but for now that decision doesn't seem like a realistic one to make.
As I alluded to earlier in this thread, I do think that the grammar becomes
"more primary" as the interface/specification over the native java
implementation. I think the language variants suffered a bit because
Gremlin was so heavily driven by Java. Thinking about Gremlin as a language
first and then determining how it fits in Java, Groovy, Python, etc. I
think will help us.
>
> Is this aimed at the gremlin 3 or 4?
>
The grammar will land in 3.5.0, but it won't do very much initially. It's
there for us to start thinking around and to start to put it to work in the
replacement of the GremlinGroovyScriptEngine - I think we would target that
for 3.5.1. I don't think about TP4 that much tbh. I tend to think that
there is a bit more to learn about Gremlin as a grammar and how to better
support non-JVM languages before then. I also think there are a number of
gaps to fill with some missing pieces to the Gremln language itself. All of
that can happen in TP3 in my mind...It remains a decent place to experiment
and learn.
>
> Cheers
> Pieter
>
> On Tue, 2021-03-16 at 15:47 -0400, Stephen Mallette wrote:
> > Here is the PR: https://github.com/apache/tinkerpop/pull/1408
> >
> > On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette
> > <sp...@gmail.com>
> > wrote:
> >
> > > No branch yet, but I think I will be sending the PR today.
> > >
> > > On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier
> > > <jo...@fortytwo.net>
> > > wrote:
> > >
> > > > Is there a branch we can take a look at before the PR is ready?
> > > >
> > > > Josh
> > > >
> > > > On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette
> > > > <sp...@gmail.com>
> > > > wrote:
> > > >
> > > > > I've been working on forming a pull request for this task. I
> > > > > don't
> > > > think IP
> > > > > Clearance is necessary as I originally did because the
> > > > > contribution is
> > > > > really just an ANTLR4 grammar file with some tests to validate
> > > > > things.
> > > > > Therefore, it's not a big body of independent code as I'd
> > > > > perhaps
> > > > initially
> > > > > envisioned. Compared to gremlint, this addition is pretty
> > > > > simple and
> > > > > straightforward. I've created this issue in JIRA with some
> > > > > additional
> > > > notes
> > > > > on what to expect in this initial body of work:
> > > > >
> > > > > https://issues.apache.org/jira/browse/TINKERPOP-2533
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette
> > > > > <sp...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Just wanted to leave an update on this thread. It was nice to
> > > > > > see some
> > > > > > support for it. I've not had time to focus on the task itself
> > > > > > so sorry
> > > > > > there hasn't been much movement, but I hope to see it on
> > > > > > track soon. I
> > > > > > thought to update the thread after I came across yet another
> > > > > > nice
> > > > usage
> > > > > for
> > > > > > it. I've long wanted to unify our test framework (i.e.
> > > > > > deprecate the
> > > > JVM
> > > > > > process suite in favor of the GLV test suite). I was
> > > > > > experimenting
> > > > with
> > > > > > what that might look like on Friday and hit a circular
> > > > > > dependency
> > > > which
> > > > > > constantly trips things up where gremlin-test wants to depend
> > > > > > on
> > > > > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy
> > > > > > depends
> > > > on
> > > > > > gremlin-test and tinkergraph with <test> scope already. I
> > > > > > think the
> > > > > > introduction of gremlin-script would let gremlin-test build
> > > > > > the
> > > > Traversal
> > > > > > object from a Gremlin string and thus avoid that circular
> > > > relationship.
> > > > > >
> > > > > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail
> > > > > > <pi...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > I have often thought the language specification should be a
> > > > > > > project
> > > > > > > separate from the implementations, and done in a formal but
> > > > > > > plain
> > > > > > > English format similar to OMG or IETF specifications.
> > > > > > >
> > > > > > > I suspect Sqlg's code base would have been fastly different
> > > > > > > if it had
> > > > > > > evolved from a grammer instead of an api.
> > > > > > >
> > > > > > > Cheers
> > > > > > > Pieter
> > > > > > >
> > > > > > > On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> > > > > > > > As a project, over the years, we've often been asked the
> > > > > > > > question
> > > > as
> > > > > > > > to why
> > > > > > > > Gremlin doesn't have an ANTLR style grammar. There have
> > > > > > > > been
> > > > varying
> > > > > > > > answers over the years to explain the reasoning but in
> > > > > > > > recent years
> > > > > > > > I've
> > > > > > > > started to see where our dependence on Java for driving
> > > > > > > > Gremlin
> > > > > > > > design has
> > > > > > > > not translated well as we have expanded Gremlin into
> > > > > > > > other
> > > > > > > > programming
> > > > > > > > ecosystems. Using Java has often allowed idioms of that
> > > > > > > > language to
> > > > > > > > leak
> > > > > > > > into Gremlin itself which introduces friction when
> > > > > > > > implemented
> > > > > > > > outside of
> > > > > > > > the JVM. I think that there is some advantage to
> > > > > > > > designing Gremlin
> > > > > > > > more
> > > > > > > > with just graphs/usage in mind and then determining how
> > > > > > > > that design
> > > > > > > > choice
> > > > > > > > looks in each programming language.
> > > > > > > >
> > > > > > > > I think that using an ANTLR grammar to drive that design
> > > > > > > > work for
> > > > > > > > Gremlin
> > > > > > > > makes a lot of sense in this context. We would
> > > > > > > > effectively have
> > > > > > > > something
> > > > > > > > like a gremlin-script which would become the new language
> > > > archetype.
> > > > > > > > New
> > > > > > > > steps, language changes, etc. would be discussed in its
> > > > > > > > context and
> > > > > > > > then
> > > > > > > > implemented in the grammar and later in each programming
> > > > > > > > language
> > > > we
> > > > > > > > support in the style a developer would expect. An
> > > > > > > > interesting
> > > > upside
> > > > > > > > of
> > > > > > > > this approach is that we can implement gremlin-script in
> > > > > > > > the
> > > > > > > > ScriptEngine
> > > > > > > > and replace GremlinGroovyScriptEngine which would help us
> > > > strengthen
> > > > > > > > our
> > > > > > > > security story in Gremlin Server. Groovy processing would
> > > > > > > > just be a
> > > > > > > > fallback to Gremlin scripts that could not be processed
> > > > > > > > by the AST.
> > > > > > > > In fact
> > > > > > > > users who didn't need Groovy could simply not install it
> > > > > > > > at all and
> > > > > > > > thus
> > > > > > > > boast a much more secure system.
> > > > > > > >
> > > > > > > > I think that inclusion of a grammar in our project is an
> > > > > > > > exciting
> > > > new
> > > > > > > > direction for us to take and will help in a variety of
> > > > > > > > areas beyond
> > > > > > > > those
> > > > > > > > I've already related.
> > > > > > > >
> > > > > > > > If we like this direction, Amazon Neptune already
> > > > > > > > maintains such a
> > > > > > > > grammar
> > > > > > > > and would be willing to contribute it to the project to
> > > > > > > > live in
> > > > open
> > > > > > > > source. The contribution would go through the same IP
> > > > > > > > Clearance
> > > > > > > > process
> > > > > > > > gremlint is going through since it was developed outside
> > > > > > > > of
> > > > > > > > TinkerPop. I'd
> > > > > > > > be happy to guide that process through if we draw to
> > > > > > > > consensus
> > > > here.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
>
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Joshua Shinavier <jo...@fortytwo.net>.
Btw. to your other questions:
I think Stephen's attitude toward TP4 is that there should be a smooth
transition, so ideally this would be for TP3, migrating to TP4 when the
time comes.
Native Java traversals are not accessible to a parser, so I do not see how
they would be impacted by gremlin-grammar or gremlin-language.
Josh
On Mon, Mar 22, 2021 at 2:00 PM Joshua Shinavier <jo...@fortytwo.net> wrote:
> I am not yet entirely sure what it means either, but I am thinking that it
> would be nice to be able to
> a) validate Gremlin expressions written in languages other than Groovy or
> Java, and
> b) parse Gremlin expressions in Gremlin language variants, producing
> JVM-based traversals, or even
> c) parse Gremlin expressions in Gremlin language variants, producing
> native traversals in the host language
> These options range from easy (a) to hard (c).
>
> Another way to look at this is: let's use Stephen's grammar as a template
> for a more generic grammar which is more flexible w.r.t. the input language.
>
> Josh
>
>
>
> On Mon, Mar 22, 2021 at 11:37 AM pieter gmail <pi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Exciting as this is I am not quite sure what it means.
>>
>> Naively perhaps it the idea,
>> Arbitary gremlin string -> antlr parser -> some AST walker -> gremlin
>> byte code -> java in memory steps ... -> voila
>>
>> Is the grammar going to be the primary and only
>> interface/specification, or will the native java implementation bypass
>> the grammar going straight to the steps instead?
>>
>> Is this aimed at the gremlin 3 or 4?
>>
>> Cheers
>> Pieter
>>
>> On Tue, 2021-03-16 at 15:47 -0400, Stephen Mallette wrote:
>> > Here is the PR: https://github.com/apache/tinkerpop/pull/1408
>> >
>> > On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette
>> > <sp...@gmail.com>
>> > wrote:
>> >
>> > > No branch yet, but I think I will be sending the PR today.
>> > >
>> > > On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier
>> > > <jo...@fortytwo.net>
>> > > wrote:
>> > >
>> > > > Is there a branch we can take a look at before the PR is ready?
>> > > >
>> > > > Josh
>> > > >
>> > > > On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette
>> > > > <sp...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > I've been working on forming a pull request for this task. I
>> > > > > don't
>> > > > think IP
>> > > > > Clearance is necessary as I originally did because the
>> > > > > contribution is
>> > > > > really just an ANTLR4 grammar file with some tests to validate
>> > > > > things.
>> > > > > Therefore, it's not a big body of independent code as I'd
>> > > > > perhaps
>> > > > initially
>> > > > > envisioned. Compared to gremlint, this addition is pretty
>> > > > > simple and
>> > > > > straightforward. I've created this issue in JIRA with some
>> > > > > additional
>> > > > notes
>> > > > > on what to expect in this initial body of work:
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/TINKERPOP-2533
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette
>> > > > > <sp...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Just wanted to leave an update on this thread. It was nice to
>> > > > > > see some
>> > > > > > support for it. I've not had time to focus on the task itself
>> > > > > > so sorry
>> > > > > > there hasn't been much movement, but I hope to see it on
>> > > > > > track soon. I
>> > > > > > thought to update the thread after I came across yet another
>> > > > > > nice
>> > > > usage
>> > > > > for
>> > > > > > it. I've long wanted to unify our test framework (i.e.
>> > > > > > deprecate the
>> > > > JVM
>> > > > > > process suite in favor of the GLV test suite). I was
>> > > > > > experimenting
>> > > > with
>> > > > > > what that might look like on Friday and hit a circular
>> > > > > > dependency
>> > > > which
>> > > > > > constantly trips things up where gremlin-test wants to depend
>> > > > > > on
>> > > > > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy
>> > > > > > depends
>> > > > on
>> > > > > > gremlin-test and tinkergraph with <test> scope already. I
>> > > > > > think the
>> > > > > > introduction of gremlin-script would let gremlin-test build
>> > > > > > the
>> > > > Traversal
>> > > > > > object from a Gremlin string and thus avoid that circular
>> > > > relationship.
>> > > > > >
>> > > > > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail
>> > > > > > <pi...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > +1
>> > > > > > >
>> > > > > > > I have often thought the language specification should be a
>> > > > > > > project
>> > > > > > > separate from the implementations, and done in a formal but
>> > > > > > > plain
>> > > > > > > English format similar to OMG or IETF specifications.
>> > > > > > >
>> > > > > > > I suspect Sqlg's code base would have been fastly different
>> > > > > > > if it had
>> > > > > > > evolved from a grammer instead of an api.
>> > > > > > >
>> > > > > > > Cheers
>> > > > > > > Pieter
>> > > > > > >
>> > > > > > > On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
>> > > > > > > > As a project, over the years, we've often been asked the
>> > > > > > > > question
>> > > > as
>> > > > > > > > to why
>> > > > > > > > Gremlin doesn't have an ANTLR style grammar. There have
>> > > > > > > > been
>> > > > varying
>> > > > > > > > answers over the years to explain the reasoning but in
>> > > > > > > > recent years
>> > > > > > > > I've
>> > > > > > > > started to see where our dependence on Java for driving
>> > > > > > > > Gremlin
>> > > > > > > > design has
>> > > > > > > > not translated well as we have expanded Gremlin into
>> > > > > > > > other
>> > > > > > > > programming
>> > > > > > > > ecosystems. Using Java has often allowed idioms of that
>> > > > > > > > language to
>> > > > > > > > leak
>> > > > > > > > into Gremlin itself which introduces friction when
>> > > > > > > > implemented
>> > > > > > > > outside of
>> > > > > > > > the JVM. I think that there is some advantage to
>> > > > > > > > designing Gremlin
>> > > > > > > > more
>> > > > > > > > with just graphs/usage in mind and then determining how
>> > > > > > > > that design
>> > > > > > > > choice
>> > > > > > > > looks in each programming language.
>> > > > > > > >
>> > > > > > > > I think that using an ANTLR grammar to drive that design
>> > > > > > > > work for
>> > > > > > > > Gremlin
>> > > > > > > > makes a lot of sense in this context. We would
>> > > > > > > > effectively have
>> > > > > > > > something
>> > > > > > > > like a gremlin-script which would become the new language
>> > > > archetype.
>> > > > > > > > New
>> > > > > > > > steps, language changes, etc. would be discussed in its
>> > > > > > > > context and
>> > > > > > > > then
>> > > > > > > > implemented in the grammar and later in each programming
>> > > > > > > > language
>> > > > we
>> > > > > > > > support in the style a developer would expect. An
>> > > > > > > > interesting
>> > > > upside
>> > > > > > > > of
>> > > > > > > > this approach is that we can implement gremlin-script in
>> > > > > > > > the
>> > > > > > > > ScriptEngine
>> > > > > > > > and replace GremlinGroovyScriptEngine which would help us
>> > > > strengthen
>> > > > > > > > our
>> > > > > > > > security story in Gremlin Server. Groovy processing would
>> > > > > > > > just be a
>> > > > > > > > fallback to Gremlin scripts that could not be processed
>> > > > > > > > by the AST.
>> > > > > > > > In fact
>> > > > > > > > users who didn't need Groovy could simply not install it
>> > > > > > > > at all and
>> > > > > > > > thus
>> > > > > > > > boast a much more secure system.
>> > > > > > > >
>> > > > > > > > I think that inclusion of a grammar in our project is an
>> > > > > > > > exciting
>> > > > new
>> > > > > > > > direction for us to take and will help in a variety of
>> > > > > > > > areas beyond
>> > > > > > > > those
>> > > > > > > > I've already related.
>> > > > > > > >
>> > > > > > > > If we like this direction, Amazon Neptune already
>> > > > > > > > maintains such a
>> > > > > > > > grammar
>> > > > > > > > and would be willing to contribute it to the project to
>> > > > > > > > live in
>> > > > open
>> > > > > > > > source. The contribution would go through the same IP
>> > > > > > > > Clearance
>> > > > > > > > process
>> > > > > > > > gremlint is going through since it was developed outside
>> > > > > > > > of
>> > > > > > > > TinkerPop. I'd
>> > > > > > > > be happy to guide that process through if we draw to
>> > > > > > > > consensus
>> > > > here.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > >
>> > > >
>> > >
>>
>>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Stephen Mallette <sp...@gmail.com>.
Some answers inline below:
On Mon, Mar 22, 2021 at 5:00 PM Joshua Shinavier <jo...@fortytwo.net> wrote:
> I am not yet entirely sure what it means either, but I am thinking that it
> would be nice to be able to
> a) validate Gremlin expressions written in languages other than Groovy or
> Java, and
>
Yes - that would be a nice practical feature.
> b) parse Gremlin expressions in Gremlin language variants, producing
> JVM-based traversals, or even
>
I like this idea too. It would be nice if Gremlin strings didn't have to be
constrained to a Groovy syntax when you're familiar with Python.
> c) parse Gremlin expressions in Gremlin language variants, producing native
> traversals in the host language
> These options range from easy (a) to hard (c).
>
This idea has long been on my mind in various ways. I'm not sure how we'd
ever reasonably try to do that, but if I were to be pressured for a
subject/theme for TP4 I think this would be it.
> Another way to look at this is: let's use Stephen's grammar as a template
> for a more generic grammar which is more flexible w.r.t. the input
> language.
>
> Josh
>
>
>
> On Mon, Mar 22, 2021 at 11:37 AM pieter gmail <pi...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Exciting as this is I am not quite sure what it means.
> >
> > Naively perhaps it the idea,
> > Arbitary gremlin string -> antlr parser -> some AST walker -> gremlin
> > byte code -> java in memory steps ... -> voila
> >
> > Is the grammar going to be the primary and only
> > interface/specification, or will the native java implementation bypass
> > the grammar going straight to the steps instead?
> >
> > Is this aimed at the gremlin 3 or 4?
> >
> > Cheers
> > Pieter
> >
> > On Tue, 2021-03-16 at 15:47 -0400, Stephen Mallette wrote:
> > > Here is the PR: https://github.com/apache/tinkerpop/pull/1408
> > >
> > > On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette
> > > <sp...@gmail.com>
> > > wrote:
> > >
> > > > No branch yet, but I think I will be sending the PR today.
> > > >
> > > > On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier
> > > > <jo...@fortytwo.net>
> > > > wrote:
> > > >
> > > > > Is there a branch we can take a look at before the PR is ready?
> > > > >
> > > > > Josh
> > > > >
> > > > > On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette
> > > > > <sp...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I've been working on forming a pull request for this task. I
> > > > > > don't
> > > > > think IP
> > > > > > Clearance is necessary as I originally did because the
> > > > > > contribution is
> > > > > > really just an ANTLR4 grammar file with some tests to validate
> > > > > > things.
> > > > > > Therefore, it's not a big body of independent code as I'd
> > > > > > perhaps
> > > > > initially
> > > > > > envisioned. Compared to gremlint, this addition is pretty
> > > > > > simple and
> > > > > > straightforward. I've created this issue in JIRA with some
> > > > > > additional
> > > > > notes
> > > > > > on what to expect in this initial body of work:
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/TINKERPOP-2533
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette
> > > > > > <sp...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Just wanted to leave an update on this thread. It was nice to
> > > > > > > see some
> > > > > > > support for it. I've not had time to focus on the task itself
> > > > > > > so sorry
> > > > > > > there hasn't been much movement, but I hope to see it on
> > > > > > > track soon. I
> > > > > > > thought to update the thread after I came across yet another
> > > > > > > nice
> > > > > usage
> > > > > > for
> > > > > > > it. I've long wanted to unify our test framework (i.e.
> > > > > > > deprecate the
> > > > > JVM
> > > > > > > process suite in favor of the GLV test suite). I was
> > > > > > > experimenting
> > > > > with
> > > > > > > what that might look like on Friday and hit a circular
> > > > > > > dependency
> > > > > which
> > > > > > > constantly trips things up where gremlin-test wants to depend
> > > > > > > on
> > > > > > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy
> > > > > > > depends
> > > > > on
> > > > > > > gremlin-test and tinkergraph with <test> scope already. I
> > > > > > > think the
> > > > > > > introduction of gremlin-script would let gremlin-test build
> > > > > > > the
> > > > > Traversal
> > > > > > > object from a Gremlin string and thus avoid that circular
> > > > > relationship.
> > > > > > >
> > > > > > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail
> > > > > > > <pi...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > I have often thought the language specification should be a
> > > > > > > > project
> > > > > > > > separate from the implementations, and done in a formal but
> > > > > > > > plain
> > > > > > > > English format similar to OMG or IETF specifications.
> > > > > > > >
> > > > > > > > I suspect Sqlg's code base would have been fastly different
> > > > > > > > if it had
> > > > > > > > evolved from a grammer instead of an api.
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > > Pieter
> > > > > > > >
> > > > > > > > On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> > > > > > > > > As a project, over the years, we've often been asked the
> > > > > > > > > question
> > > > > as
> > > > > > > > > to why
> > > > > > > > > Gremlin doesn't have an ANTLR style grammar. There have
> > > > > > > > > been
> > > > > varying
> > > > > > > > > answers over the years to explain the reasoning but in
> > > > > > > > > recent years
> > > > > > > > > I've
> > > > > > > > > started to see where our dependence on Java for driving
> > > > > > > > > Gremlin
> > > > > > > > > design has
> > > > > > > > > not translated well as we have expanded Gremlin into
> > > > > > > > > other
> > > > > > > > > programming
> > > > > > > > > ecosystems. Using Java has often allowed idioms of that
> > > > > > > > > language to
> > > > > > > > > leak
> > > > > > > > > into Gremlin itself which introduces friction when
> > > > > > > > > implemented
> > > > > > > > > outside of
> > > > > > > > > the JVM. I think that there is some advantage to
> > > > > > > > > designing Gremlin
> > > > > > > > > more
> > > > > > > > > with just graphs/usage in mind and then determining how
> > > > > > > > > that design
> > > > > > > > > choice
> > > > > > > > > looks in each programming language.
> > > > > > > > >
> > > > > > > > > I think that using an ANTLR grammar to drive that design
> > > > > > > > > work for
> > > > > > > > > Gremlin
> > > > > > > > > makes a lot of sense in this context. We would
> > > > > > > > > effectively have
> > > > > > > > > something
> > > > > > > > > like a gremlin-script which would become the new language
> > > > > archetype.
> > > > > > > > > New
> > > > > > > > > steps, language changes, etc. would be discussed in its
> > > > > > > > > context and
> > > > > > > > > then
> > > > > > > > > implemented in the grammar and later in each programming
> > > > > > > > > language
> > > > > we
> > > > > > > > > support in the style a developer would expect. An
> > > > > > > > > interesting
> > > > > upside
> > > > > > > > > of
> > > > > > > > > this approach is that we can implement gremlin-script in
> > > > > > > > > the
> > > > > > > > > ScriptEngine
> > > > > > > > > and replace GremlinGroovyScriptEngine which would help us
> > > > > strengthen
> > > > > > > > > our
> > > > > > > > > security story in Gremlin Server. Groovy processing would
> > > > > > > > > just be a
> > > > > > > > > fallback to Gremlin scripts that could not be processed
> > > > > > > > > by the AST.
> > > > > > > > > In fact
> > > > > > > > > users who didn't need Groovy could simply not install it
> > > > > > > > > at all and
> > > > > > > > > thus
> > > > > > > > > boast a much more secure system.
> > > > > > > > >
> > > > > > > > > I think that inclusion of a grammar in our project is an
> > > > > > > > > exciting
> > > > > new
> > > > > > > > > direction for us to take and will help in a variety of
> > > > > > > > > areas beyond
> > > > > > > > > those
> > > > > > > > > I've already related.
> > > > > > > > >
> > > > > > > > > If we like this direction, Amazon Neptune already
> > > > > > > > > maintains such a
> > > > > > > > > grammar
> > > > > > > > > and would be willing to contribute it to the project to
> > > > > > > > > live in
> > > > > open
> > > > > > > > > source. The contribution would go through the same IP
> > > > > > > > > Clearance
> > > > > > > > > process
> > > > > > > > > gremlint is going through since it was developed outside
> > > > > > > > > of
> > > > > > > > > TinkerPop. I'd
> > > > > > > > > be happy to guide that process through if we draw to
> > > > > > > > > consensus
> > > > > here.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> >
> >
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Joshua Shinavier <jo...@fortytwo.net>.
I am not yet entirely sure what it means either, but I am thinking that it
would be nice to be able to
a) validate Gremlin expressions written in languages other than Groovy or
Java, and
b) parse Gremlin expressions in Gremlin language variants, producing
JVM-based traversals, or even
c) parse Gremlin expressions in Gremlin language variants, producing native
traversals in the host language
These options range from easy (a) to hard (c).
Another way to look at this is: let's use Stephen's grammar as a template
for a more generic grammar which is more flexible w.r.t. the input language.
Josh
On Mon, Mar 22, 2021 at 11:37 AM pieter gmail <pi...@gmail.com>
wrote:
> Hi,
>
> Exciting as this is I am not quite sure what it means.
>
> Naively perhaps it the idea,
> Arbitary gremlin string -> antlr parser -> some AST walker -> gremlin
> byte code -> java in memory steps ... -> voila
>
> Is the grammar going to be the primary and only
> interface/specification, or will the native java implementation bypass
> the grammar going straight to the steps instead?
>
> Is this aimed at the gremlin 3 or 4?
>
> Cheers
> Pieter
>
> On Tue, 2021-03-16 at 15:47 -0400, Stephen Mallette wrote:
> > Here is the PR: https://github.com/apache/tinkerpop/pull/1408
> >
> > On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette
> > <sp...@gmail.com>
> > wrote:
> >
> > > No branch yet, but I think I will be sending the PR today.
> > >
> > > On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier
> > > <jo...@fortytwo.net>
> > > wrote:
> > >
> > > > Is there a branch we can take a look at before the PR is ready?
> > > >
> > > > Josh
> > > >
> > > > On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette
> > > > <sp...@gmail.com>
> > > > wrote:
> > > >
> > > > > I've been working on forming a pull request for this task. I
> > > > > don't
> > > > think IP
> > > > > Clearance is necessary as I originally did because the
> > > > > contribution is
> > > > > really just an ANTLR4 grammar file with some tests to validate
> > > > > things.
> > > > > Therefore, it's not a big body of independent code as I'd
> > > > > perhaps
> > > > initially
> > > > > envisioned. Compared to gremlint, this addition is pretty
> > > > > simple and
> > > > > straightforward. I've created this issue in JIRA with some
> > > > > additional
> > > > notes
> > > > > on what to expect in this initial body of work:
> > > > >
> > > > > https://issues.apache.org/jira/browse/TINKERPOP-2533
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette
> > > > > <sp...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Just wanted to leave an update on this thread. It was nice to
> > > > > > see some
> > > > > > support for it. I've not had time to focus on the task itself
> > > > > > so sorry
> > > > > > there hasn't been much movement, but I hope to see it on
> > > > > > track soon. I
> > > > > > thought to update the thread after I came across yet another
> > > > > > nice
> > > > usage
> > > > > for
> > > > > > it. I've long wanted to unify our test framework (i.e.
> > > > > > deprecate the
> > > > JVM
> > > > > > process suite in favor of the GLV test suite). I was
> > > > > > experimenting
> > > > with
> > > > > > what that might look like on Friday and hit a circular
> > > > > > dependency
> > > > which
> > > > > > constantly trips things up where gremlin-test wants to depend
> > > > > > on
> > > > > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy
> > > > > > depends
> > > > on
> > > > > > gremlin-test and tinkergraph with <test> scope already. I
> > > > > > think the
> > > > > > introduction of gremlin-script would let gremlin-test build
> > > > > > the
> > > > Traversal
> > > > > > object from a Gremlin string and thus avoid that circular
> > > > relationship.
> > > > > >
> > > > > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail
> > > > > > <pi...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > I have often thought the language specification should be a
> > > > > > > project
> > > > > > > separate from the implementations, and done in a formal but
> > > > > > > plain
> > > > > > > English format similar to OMG or IETF specifications.
> > > > > > >
> > > > > > > I suspect Sqlg's code base would have been fastly different
> > > > > > > if it had
> > > > > > > evolved from a grammer instead of an api.
> > > > > > >
> > > > > > > Cheers
> > > > > > > Pieter
> > > > > > >
> > > > > > > On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> > > > > > > > As a project, over the years, we've often been asked the
> > > > > > > > question
> > > > as
> > > > > > > > to why
> > > > > > > > Gremlin doesn't have an ANTLR style grammar. There have
> > > > > > > > been
> > > > varying
> > > > > > > > answers over the years to explain the reasoning but in
> > > > > > > > recent years
> > > > > > > > I've
> > > > > > > > started to see where our dependence on Java for driving
> > > > > > > > Gremlin
> > > > > > > > design has
> > > > > > > > not translated well as we have expanded Gremlin into
> > > > > > > > other
> > > > > > > > programming
> > > > > > > > ecosystems. Using Java has often allowed idioms of that
> > > > > > > > language to
> > > > > > > > leak
> > > > > > > > into Gremlin itself which introduces friction when
> > > > > > > > implemented
> > > > > > > > outside of
> > > > > > > > the JVM. I think that there is some advantage to
> > > > > > > > designing Gremlin
> > > > > > > > more
> > > > > > > > with just graphs/usage in mind and then determining how
> > > > > > > > that design
> > > > > > > > choice
> > > > > > > > looks in each programming language.
> > > > > > > >
> > > > > > > > I think that using an ANTLR grammar to drive that design
> > > > > > > > work for
> > > > > > > > Gremlin
> > > > > > > > makes a lot of sense in this context. We would
> > > > > > > > effectively have
> > > > > > > > something
> > > > > > > > like a gremlin-script which would become the new language
> > > > archetype.
> > > > > > > > New
> > > > > > > > steps, language changes, etc. would be discussed in its
> > > > > > > > context and
> > > > > > > > then
> > > > > > > > implemented in the grammar and later in each programming
> > > > > > > > language
> > > > we
> > > > > > > > support in the style a developer would expect. An
> > > > > > > > interesting
> > > > upside
> > > > > > > > of
> > > > > > > > this approach is that we can implement gremlin-script in
> > > > > > > > the
> > > > > > > > ScriptEngine
> > > > > > > > and replace GremlinGroovyScriptEngine which would help us
> > > > strengthen
> > > > > > > > our
> > > > > > > > security story in Gremlin Server. Groovy processing would
> > > > > > > > just be a
> > > > > > > > fallback to Gremlin scripts that could not be processed
> > > > > > > > by the AST.
> > > > > > > > In fact
> > > > > > > > users who didn't need Groovy could simply not install it
> > > > > > > > at all and
> > > > > > > > thus
> > > > > > > > boast a much more secure system.
> > > > > > > >
> > > > > > > > I think that inclusion of a grammar in our project is an
> > > > > > > > exciting
> > > > new
> > > > > > > > direction for us to take and will help in a variety of
> > > > > > > > areas beyond
> > > > > > > > those
> > > > > > > > I've already related.
> > > > > > > >
> > > > > > > > If we like this direction, Amazon Neptune already
> > > > > > > > maintains such a
> > > > > > > > grammar
> > > > > > > > and would be willing to contribute it to the project to
> > > > > > > > live in
> > > > open
> > > > > > > > source. The contribution would go through the same IP
> > > > > > > > Clearance
> > > > > > > > process
> > > > > > > > gremlint is going through since it was developed outside
> > > > > > > > of
> > > > > > > > TinkerPop. I'd
> > > > > > > > be happy to guide that process through if we draw to
> > > > > > > > consensus
> > > > here.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
>
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by pieter gmail <pi...@gmail.com>.
Hi,
Exciting as this is I am not quite sure what it means.
Naively perhaps it the idea,
Arbitary gremlin string -> antlr parser -> some AST walker -> gremlin
byte code -> java in memory steps ... -> voila
Is the grammar going to be the primary and only
interface/specification, or will the native java implementation bypass
the grammar going straight to the steps instead?
Is this aimed at the gremlin 3 or 4?
Cheers
Pieter
On Tue, 2021-03-16 at 15:47 -0400, Stephen Mallette wrote:
> Here is the PR: https://github.com/apache/tinkerpop/pull/1408
>
> On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette
> <sp...@gmail.com>
> wrote:
>
> > No branch yet, but I think I will be sending the PR today.
> >
> > On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier
> > <jo...@fortytwo.net>
> > wrote:
> >
> > > Is there a branch we can take a look at before the PR is ready?
> > >
> > > Josh
> > >
> > > On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette
> > > <sp...@gmail.com>
> > > wrote:
> > >
> > > > I've been working on forming a pull request for this task. I
> > > > don't
> > > think IP
> > > > Clearance is necessary as I originally did because the
> > > > contribution is
> > > > really just an ANTLR4 grammar file with some tests to validate
> > > > things.
> > > > Therefore, it's not a big body of independent code as I'd
> > > > perhaps
> > > initially
> > > > envisioned. Compared to gremlint, this addition is pretty
> > > > simple and
> > > > straightforward. I've created this issue in JIRA with some
> > > > additional
> > > notes
> > > > on what to expect in this initial body of work:
> > > >
> > > > https://issues.apache.org/jira/browse/TINKERPOP-2533
> > > >
> > > >
> > > >
> > > > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette
> > > > <sp...@gmail.com>
> > > > wrote:
> > > >
> > > > > Just wanted to leave an update on this thread. It was nice to
> > > > > see some
> > > > > support for it. I've not had time to focus on the task itself
> > > > > so sorry
> > > > > there hasn't been much movement, but I hope to see it on
> > > > > track soon. I
> > > > > thought to update the thread after I came across yet another
> > > > > nice
> > > usage
> > > > for
> > > > > it. I've long wanted to unify our test framework (i.e.
> > > > > deprecate the
> > > JVM
> > > > > process suite in favor of the GLV test suite). I was
> > > > > experimenting
> > > with
> > > > > what that might look like on Friday and hit a circular
> > > > > dependency
> > > which
> > > > > constantly trips things up where gremlin-test wants to depend
> > > > > on
> > > > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy
> > > > > depends
> > > on
> > > > > gremlin-test and tinkergraph with <test> scope already. I
> > > > > think the
> > > > > introduction of gremlin-script would let gremlin-test build
> > > > > the
> > > Traversal
> > > > > object from a Gremlin string and thus avoid that circular
> > > relationship.
> > > > >
> > > > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail
> > > > > <pi...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > I have often thought the language specification should be a
> > > > > > project
> > > > > > separate from the implementations, and done in a formal but
> > > > > > plain
> > > > > > English format similar to OMG or IETF specifications.
> > > > > >
> > > > > > I suspect Sqlg's code base would have been fastly different
> > > > > > if it had
> > > > > > evolved from a grammer instead of an api.
> > > > > >
> > > > > > Cheers
> > > > > > Pieter
> > > > > >
> > > > > > On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> > > > > > > As a project, over the years, we've often been asked the
> > > > > > > question
> > > as
> > > > > > > to why
> > > > > > > Gremlin doesn't have an ANTLR style grammar. There have
> > > > > > > been
> > > varying
> > > > > > > answers over the years to explain the reasoning but in
> > > > > > > recent years
> > > > > > > I've
> > > > > > > started to see where our dependence on Java for driving
> > > > > > > Gremlin
> > > > > > > design has
> > > > > > > not translated well as we have expanded Gremlin into
> > > > > > > other
> > > > > > > programming
> > > > > > > ecosystems. Using Java has often allowed idioms of that
> > > > > > > language to
> > > > > > > leak
> > > > > > > into Gremlin itself which introduces friction when
> > > > > > > implemented
> > > > > > > outside of
> > > > > > > the JVM. I think that there is some advantage to
> > > > > > > designing Gremlin
> > > > > > > more
> > > > > > > with just graphs/usage in mind and then determining how
> > > > > > > that design
> > > > > > > choice
> > > > > > > looks in each programming language.
> > > > > > >
> > > > > > > I think that using an ANTLR grammar to drive that design
> > > > > > > work for
> > > > > > > Gremlin
> > > > > > > makes a lot of sense in this context. We would
> > > > > > > effectively have
> > > > > > > something
> > > > > > > like a gremlin-script which would become the new language
> > > archetype.
> > > > > > > New
> > > > > > > steps, language changes, etc. would be discussed in its
> > > > > > > context and
> > > > > > > then
> > > > > > > implemented in the grammar and later in each programming
> > > > > > > language
> > > we
> > > > > > > support in the style a developer would expect. An
> > > > > > > interesting
> > > upside
> > > > > > > of
> > > > > > > this approach is that we can implement gremlin-script in
> > > > > > > the
> > > > > > > ScriptEngine
> > > > > > > and replace GremlinGroovyScriptEngine which would help us
> > > strengthen
> > > > > > > our
> > > > > > > security story in Gremlin Server. Groovy processing would
> > > > > > > just be a
> > > > > > > fallback to Gremlin scripts that could not be processed
> > > > > > > by the AST.
> > > > > > > In fact
> > > > > > > users who didn't need Groovy could simply not install it
> > > > > > > at all and
> > > > > > > thus
> > > > > > > boast a much more secure system.
> > > > > > >
> > > > > > > I think that inclusion of a grammar in our project is an
> > > > > > > exciting
> > > new
> > > > > > > direction for us to take and will help in a variety of
> > > > > > > areas beyond
> > > > > > > those
> > > > > > > I've already related.
> > > > > > >
> > > > > > > If we like this direction, Amazon Neptune already
> > > > > > > maintains such a
> > > > > > > grammar
> > > > > > > and would be willing to contribute it to the project to
> > > > > > > live in
> > > open
> > > > > > > source. The contribution would go through the same IP
> > > > > > > Clearance
> > > > > > > process
> > > > > > > gremlint is going through since it was developed outside
> > > > > > > of
> > > > > > > TinkerPop. I'd
> > > > > > > be happy to guide that process through if we draw to
> > > > > > > consensus
> > > here.
> > > > > >
> > > > > >
> > > > > >
> > > >
> > >
> >
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Stephen Mallette <sp...@gmail.com>.
I've renamed it to gremlin-language - I plan to merge this PR today. Thanks!
On Thu, Mar 25, 2021 at 11:27 AM Joshua Shinavier <jo...@fortytwo.net> wrote:
> Hi Stephen,
>
> Replies inline.
>
> On Wed, Mar 24, 2021 at 5:18 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > [...]
> > I'd thought that it made sense that Groovy and the grammar stayed
> > fairly close to one another as a nice point of Groovy is that you can
> make
> > it do neat tricks when building DSLs but I don't think it necessarily
> needs
> > to.
> >
>
> I think those neat tricks ought to be possible with the generalized grammar
> as well, but it's all a little hand-wavy right now. If this is of interest,
> which it seems to be, I'll see if I can do a small proof of concept some
> time soon.
>
>
>
> > 1. Specify an abstract Gremlin grammar in a neutral language like YAML
> > > 2. Write some helper code for generating ANTLR grammars from the
> YAML
> > > 3. For each Gremlin language variant, write a smaller amount of code
> > > based on (2) to generate a language-specific ANTLR grammar
> > >
> >
> > I like how you present this idea because one of our big problems and big
> > assets are language variants and generating our way to ANTLR support in
> > each makes sense to me. I'd be curious what a YAML representation and the
> > related work might look like and how you think we might structure it.
> >
>
>
> I'm curious too, but it would probably look a lot like the other kinds of
> transformations we perform on schemas and data with YAML or JSON as the
> source. E.g. instead of an expression like this:
>
> traversalMethod_out:
> 'out' LPAREN stringLiteralList RPAREN
> ;
>
> you'd have something more like this:
>
> - name: out
> category: traversalMethod
> parameters:
> - type:
> list: label
>
> and we would transform the latter to the former in the case of
> Gremlin-Groovy, and to something slightly different in the case of other
> variants.
>
>
>
>
> > Are you suggesting I change gremlin-grammar to gremlin-language with that
> > work in mind?
>
>
>
> That's my suggestion, because gremlin-language sounds a little more
> inclusive of other abstract Gremlin language specifications in addition to
> the grammar. Not to get too far ahead of ourselves, but I think it should
> be possible to use the YAML-based specification I'm hinting at above for
> other things like serializing/deserializing Gremlin traversals, and
> traversal results in non-text formats like JSON, Thrift, Protobuf, Avro,
> YAML of course, etc. A cleaner solution to the "format zoo" was another
> pain area we discussed last year, where I think mappings can help.
>
>
>
> > I don't think it's a problem to do so. I guess
> > gremlin-language would eventually house (1), (2), and (3)? Right now,
> > though, gremlin-grammar also generates a Java parser from the one ANTLR
> > file I have there. Would the idea be that we'd organize this module to
> > generate parsers for each language we supported as well? or would that
> live
> > somewhere else?
> >
>
> Yeah, I think once we can generate the parser for Gremlin-Groovy (which
> should look almost identical to what you already have), it ought to be
> straightforward to generate parser for other languages. Btw. if I give this
> a go, and the solution looks promising, maybe I will make it the topic of a
> future Category Theory and Applications session. The first session on
> Dragon is tonight at 6pm PDT.
>
> Josh
>
>
>
>
> >
> >
> > >
> > > Btw. I will be giving a Category Theory and Applications presentation
> > > <https://www.meetup.com/Category-Theory/events/nnrhgsyccfbhc/> next
> week
> > > which will illustrate how something like the above might work.
> > >
> > > Josh
> > >
> > >
> > >
> > > On Tue, Mar 16, 2021 at 12:48 PM Stephen Mallette <
> spmallette@gmail.com>
> > > wrote:
> > >
> > > > Here is the PR: https://github.com/apache/tinkerpop/pull/1408
> > > >
> > > > On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette <
> spmallette@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > No branch yet, but I think I will be sending the PR today.
> > > > >
> > > > > On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier <
> josh@fortytwo.net>
> > > > > wrote:
> > > > >
> > > > >> Is there a branch we can take a look at before the PR is ready?
> > > > >>
> > > > >> Josh
> > > > >>
> > > > >> On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette <
> > > spmallette@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > I've been working on forming a pull request for this task. I
> don't
> > > > >> think IP
> > > > >> > Clearance is necessary as I originally did because the
> > contribution
> > > is
> > > > >> > really just an ANTLR4 grammar file with some tests to validate
> > > things.
> > > > >> > Therefore, it's not a big body of independent code as I'd
> perhaps
> > > > >> initially
> > > > >> > envisioned. Compared to gremlint, this addition is pretty simple
> > and
> > > > >> > straightforward. I've created this issue in JIRA with some
> > > additional
> > > > >> notes
> > > > >> > on what to expect in this initial body of work:
> > > > >> >
> > > > >> > https://issues.apache.org/jira/browse/TINKERPOP-2533
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette <
> > > > spmallette@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Just wanted to leave an update on this thread. It was nice to
> > see
> > > > some
> > > > >> > > support for it. I've not had time to focus on the task itself
> so
> > > > sorry
> > > > >> > > there hasn't been much movement, but I hope to see it on track
> > > > soon. I
> > > > >> > > thought to update the thread after I came across yet another
> > nice
> > > > >> usage
> > > > >> > for
> > > > >> > > it. I've long wanted to unify our test framework (i.e.
> deprecate
> > > the
> > > > >> JVM
> > > > >> > > process suite in favor of the GLV test suite). I was
> > experimenting
> > > > >> with
> > > > >> > > what that might look like on Friday and hit a circular
> > dependency
> > > > >> which
> > > > >> > > constantly trips things up where gremlin-test wants to depend
> on
> > > > >> > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy
> > > depends
> > > > >> on
> > > > >> > > gremlin-test and tinkergraph with <test> scope already. I
> think
> > > the
> > > > >> > > introduction of gremlin-script would let gremlin-test build
> the
> > > > >> Traversal
> > > > >> > > object from a Gremlin string and thus avoid that circular
> > > > >> relationship.
> > > > >> > >
> > > > >> > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail <
> > > > pieter.martin@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> +1
> > > > >> > >>
> > > > >> > >> I have often thought the language specification should be a
> > > project
> > > > >> > >> separate from the implementations, and done in a formal but
> > plain
> > > > >> > >> English format similar to OMG or IETF specifications.
> > > > >> > >>
> > > > >> > >> I suspect Sqlg's code base would have been fastly different
> if
> > it
> > > > had
> > > > >> > >> evolved from a grammer instead of an api.
> > > > >> > >>
> > > > >> > >> Cheers
> > > > >> > >> Pieter
> > > > >> > >>
> > > > >> > >> On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> > > > >> > >> > As a project, over the years, we've often been asked the
> > > question
> > > > >> as
> > > > >> > >> > to why
> > > > >> > >> > Gremlin doesn't have an ANTLR style grammar. There have
> been
> > > > >> varying
> > > > >> > >> > answers over the years to explain the reasoning but in
> recent
> > > > years
> > > > >> > >> > I've
> > > > >> > >> > started to see where our dependence on Java for driving
> > Gremlin
> > > > >> > >> > design has
> > > > >> > >> > not translated well as we have expanded Gremlin into other
> > > > >> > >> > programming
> > > > >> > >> > ecosystems. Using Java has often allowed idioms of that
> > > language
> > > > to
> > > > >> > >> > leak
> > > > >> > >> > into Gremlin itself which introduces friction when
> > implemented
> > > > >> > >> > outside of
> > > > >> > >> > the JVM. I think that there is some advantage to designing
> > > > Gremlin
> > > > >> > >> > more
> > > > >> > >> > with just graphs/usage in mind and then determining how
> that
> > > > design
> > > > >> > >> > choice
> > > > >> > >> > looks in each programming language.
> > > > >> > >> >
> > > > >> > >> > I think that using an ANTLR grammar to drive that design
> work
> > > for
> > > > >> > >> > Gremlin
> > > > >> > >> > makes a lot of sense in this context. We would effectively
> > have
> > > > >> > >> > something
> > > > >> > >> > like a gremlin-script which would become the new language
> > > > >> archetype.
> > > > >> > >> > New
> > > > >> > >> > steps, language changes, etc. would be discussed in its
> > context
> > > > and
> > > > >> > >> > then
> > > > >> > >> > implemented in the grammar and later in each programming
> > > language
> > > > >> we
> > > > >> > >> > support in the style a developer would expect. An
> interesting
> > > > >> upside
> > > > >> > >> > of
> > > > >> > >> > this approach is that we can implement gremlin-script in
> the
> > > > >> > >> > ScriptEngine
> > > > >> > >> > and replace GremlinGroovyScriptEngine which would help us
> > > > >> strengthen
> > > > >> > >> > our
> > > > >> > >> > security story in Gremlin Server. Groovy processing would
> > just
> > > > be a
> > > > >> > >> > fallback to Gremlin scripts that could not be processed by
> > the
> > > > AST.
> > > > >> > >> > In fact
> > > > >> > >> > users who didn't need Groovy could simply not install it at
> > all
> > > > and
> > > > >> > >> > thus
> > > > >> > >> > boast a much more secure system.
> > > > >> > >> >
> > > > >> > >> > I think that inclusion of a grammar in our project is an
> > > exciting
> > > > >> new
> > > > >> > >> > direction for us to take and will help in a variety of
> areas
> > > > beyond
> > > > >> > >> > those
> > > > >> > >> > I've already related.
> > > > >> > >> >
> > > > >> > >> > If we like this direction, Amazon Neptune already maintains
> > > such
> > > > a
> > > > >> > >> > grammar
> > > > >> > >> > and would be willing to contribute it to the project to
> live
> > in
> > > > >> open
> > > > >> > >> > source. The contribution would go through the same IP
> > Clearance
> > > > >> > >> > process
> > > > >> > >> > gremlint is going through since it was developed outside of
> > > > >> > >> > TinkerPop. I'd
> > > > >> > >> > be happy to guide that process through if we draw to
> > consensus
> > > > >> here.
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Joshua Shinavier <jo...@fortytwo.net>.
Hi Stephen,
Replies inline.
On Wed, Mar 24, 2021 at 5:18 AM Stephen Mallette <sp...@gmail.com>
wrote:
> [...]
> I'd thought that it made sense that Groovy and the grammar stayed
> fairly close to one another as a nice point of Groovy is that you can make
> it do neat tricks when building DSLs but I don't think it necessarily needs
> to.
>
I think those neat tricks ought to be possible with the generalized grammar
as well, but it's all a little hand-wavy right now. If this is of interest,
which it seems to be, I'll see if I can do a small proof of concept some
time soon.
> 1. Specify an abstract Gremlin grammar in a neutral language like YAML
> > 2. Write some helper code for generating ANTLR grammars from the YAML
> > 3. For each Gremlin language variant, write a smaller amount of code
> > based on (2) to generate a language-specific ANTLR grammar
> >
>
> I like how you present this idea because one of our big problems and big
> assets are language variants and generating our way to ANTLR support in
> each makes sense to me. I'd be curious what a YAML representation and the
> related work might look like and how you think we might structure it.
>
I'm curious too, but it would probably look a lot like the other kinds of
transformations we perform on schemas and data with YAML or JSON as the
source. E.g. instead of an expression like this:
traversalMethod_out:
'out' LPAREN stringLiteralList RPAREN
;
you'd have something more like this:
- name: out
category: traversalMethod
parameters:
- type:
list: label
and we would transform the latter to the former in the case of
Gremlin-Groovy, and to something slightly different in the case of other
variants.
> Are you suggesting I change gremlin-grammar to gremlin-language with that
> work in mind?
That's my suggestion, because gremlin-language sounds a little more
inclusive of other abstract Gremlin language specifications in addition to
the grammar. Not to get too far ahead of ourselves, but I think it should
be possible to use the YAML-based specification I'm hinting at above for
other things like serializing/deserializing Gremlin traversals, and
traversal results in non-text formats like JSON, Thrift, Protobuf, Avro,
YAML of course, etc. A cleaner solution to the "format zoo" was another
pain area we discussed last year, where I think mappings can help.
> I don't think it's a problem to do so. I guess
> gremlin-language would eventually house (1), (2), and (3)? Right now,
> though, gremlin-grammar also generates a Java parser from the one ANTLR
> file I have there. Would the idea be that we'd organize this module to
> generate parsers for each language we supported as well? or would that live
> somewhere else?
>
Yeah, I think once we can generate the parser for Gremlin-Groovy (which
should look almost identical to what you already have), it ought to be
straightforward to generate parser for other languages. Btw. if I give this
a go, and the solution looks promising, maybe I will make it the topic of a
future Category Theory and Applications session. The first session on
Dragon is tonight at 6pm PDT.
Josh
>
>
> >
> > Btw. I will be giving a Category Theory and Applications presentation
> > <https://www.meetup.com/Category-Theory/events/nnrhgsyccfbhc/> next week
> > which will illustrate how something like the above might work.
> >
> > Josh
> >
> >
> >
> > On Tue, Mar 16, 2021 at 12:48 PM Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > Here is the PR: https://github.com/apache/tinkerpop/pull/1408
> > >
> > > On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette <spmallette@gmail.com
> >
> > > wrote:
> > >
> > > > No branch yet, but I think I will be sending the PR today.
> > > >
> > > > On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier <jo...@fortytwo.net>
> > > > wrote:
> > > >
> > > >> Is there a branch we can take a look at before the PR is ready?
> > > >>
> > > >> Josh
> > > >>
> > > >> On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette <
> > spmallette@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > I've been working on forming a pull request for this task. I don't
> > > >> think IP
> > > >> > Clearance is necessary as I originally did because the
> contribution
> > is
> > > >> > really just an ANTLR4 grammar file with some tests to validate
> > things.
> > > >> > Therefore, it's not a big body of independent code as I'd perhaps
> > > >> initially
> > > >> > envisioned. Compared to gremlint, this addition is pretty simple
> and
> > > >> > straightforward. I've created this issue in JIRA with some
> > additional
> > > >> notes
> > > >> > on what to expect in this initial body of work:
> > > >> >
> > > >> > https://issues.apache.org/jira/browse/TINKERPOP-2533
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette <
> > > spmallette@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Just wanted to leave an update on this thread. It was nice to
> see
> > > some
> > > >> > > support for it. I've not had time to focus on the task itself so
> > > sorry
> > > >> > > there hasn't been much movement, but I hope to see it on track
> > > soon. I
> > > >> > > thought to update the thread after I came across yet another
> nice
> > > >> usage
> > > >> > for
> > > >> > > it. I've long wanted to unify our test framework (i.e. deprecate
> > the
> > > >> JVM
> > > >> > > process suite in favor of the GLV test suite). I was
> experimenting
> > > >> with
> > > >> > > what that might look like on Friday and hit a circular
> dependency
> > > >> which
> > > >> > > constantly trips things up where gremlin-test wants to depend on
> > > >> > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy
> > depends
> > > >> on
> > > >> > > gremlin-test and tinkergraph with <test> scope already. I think
> > the
> > > >> > > introduction of gremlin-script would let gremlin-test build the
> > > >> Traversal
> > > >> > > object from a Gremlin string and thus avoid that circular
> > > >> relationship.
> > > >> > >
> > > >> > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail <
> > > pieter.martin@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > >> +1
> > > >> > >>
> > > >> > >> I have often thought the language specification should be a
> > project
> > > >> > >> separate from the implementations, and done in a formal but
> plain
> > > >> > >> English format similar to OMG or IETF specifications.
> > > >> > >>
> > > >> > >> I suspect Sqlg's code base would have been fastly different if
> it
> > > had
> > > >> > >> evolved from a grammer instead of an api.
> > > >> > >>
> > > >> > >> Cheers
> > > >> > >> Pieter
> > > >> > >>
> > > >> > >> On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> > > >> > >> > As a project, over the years, we've often been asked the
> > question
> > > >> as
> > > >> > >> > to why
> > > >> > >> > Gremlin doesn't have an ANTLR style grammar. There have been
> > > >> varying
> > > >> > >> > answers over the years to explain the reasoning but in recent
> > > years
> > > >> > >> > I've
> > > >> > >> > started to see where our dependence on Java for driving
> Gremlin
> > > >> > >> > design has
> > > >> > >> > not translated well as we have expanded Gremlin into other
> > > >> > >> > programming
> > > >> > >> > ecosystems. Using Java has often allowed idioms of that
> > language
> > > to
> > > >> > >> > leak
> > > >> > >> > into Gremlin itself which introduces friction when
> implemented
> > > >> > >> > outside of
> > > >> > >> > the JVM. I think that there is some advantage to designing
> > > Gremlin
> > > >> > >> > more
> > > >> > >> > with just graphs/usage in mind and then determining how that
> > > design
> > > >> > >> > choice
> > > >> > >> > looks in each programming language.
> > > >> > >> >
> > > >> > >> > I think that using an ANTLR grammar to drive that design work
> > for
> > > >> > >> > Gremlin
> > > >> > >> > makes a lot of sense in this context. We would effectively
> have
> > > >> > >> > something
> > > >> > >> > like a gremlin-script which would become the new language
> > > >> archetype.
> > > >> > >> > New
> > > >> > >> > steps, language changes, etc. would be discussed in its
> context
> > > and
> > > >> > >> > then
> > > >> > >> > implemented in the grammar and later in each programming
> > language
> > > >> we
> > > >> > >> > support in the style a developer would expect. An interesting
> > > >> upside
> > > >> > >> > of
> > > >> > >> > this approach is that we can implement gremlin-script in the
> > > >> > >> > ScriptEngine
> > > >> > >> > and replace GremlinGroovyScriptEngine which would help us
> > > >> strengthen
> > > >> > >> > our
> > > >> > >> > security story in Gremlin Server. Groovy processing would
> just
> > > be a
> > > >> > >> > fallback to Gremlin scripts that could not be processed by
> the
> > > AST.
> > > >> > >> > In fact
> > > >> > >> > users who didn't need Groovy could simply not install it at
> all
> > > and
> > > >> > >> > thus
> > > >> > >> > boast a much more secure system.
> > > >> > >> >
> > > >> > >> > I think that inclusion of a grammar in our project is an
> > exciting
> > > >> new
> > > >> > >> > direction for us to take and will help in a variety of areas
> > > beyond
> > > >> > >> > those
> > > >> > >> > I've already related.
> > > >> > >> >
> > > >> > >> > If we like this direction, Amazon Neptune already maintains
> > such
> > > a
> > > >> > >> > grammar
> > > >> > >> > and would be willing to contribute it to the project to live
> in
> > > >> open
> > > >> > >> > source. The contribution would go through the same IP
> Clearance
> > > >> > >> > process
> > > >> > >> > gremlint is going through since it was developed outside of
> > > >> > >> > TinkerPop. I'd
> > > >> > >> > be happy to guide that process through if we draw to
> consensus
> > > >> here.
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Stephen Mallette <sp...@gmail.com>.
Please see below:
On Mon, Mar 22, 2021 at 10:25 AM Joshua Shinavier <jo...@fortytwo.net> wrote:
> Awesome. This definitely fills a major gap, and as Florian said, it is
> great that it is already improving the documentation.
>
> High level-question: do you have any thoughts about Gremlin syntax
> validation *in general*, i.e. for language variants other than Groovy? It
> would be interesting to explore the following:
>
The grammar as it is tends to be roughly formed around what can be
expressed in Groovy as it is meant to be a safe way to parse Gremlin
scripts which up until now have been Groovy-based. We won't talk about
gremlin-jython :) So, tactically speaking, there's like this immediate need
to "fix groovy" but then there is this bigger picture in deciding what
direction the grammar goes from here. In thinking this contribution
through, I'd thought that it made sense that Groovy and the grammar stayed
fairly close to one another as a nice point of Groovy is that you can make
it do neat tricks when building DSLs but I don't think it necessarily needs
to.
> 1. Specify an abstract Gremlin grammar in a neutral language like YAML
> 2. Write some helper code for generating ANTLR grammars from the YAML
> 3. For each Gremlin language variant, write a smaller amount of code
> based on (2) to generate a language-specific ANTLR grammar
>
I like how you present this idea because one of our big problems and big
assets are language variants and generating our way to ANTLR support in
each makes sense to me. I'd be curious what a YAML representation and the
related work might look like and how you think we might structure it.
>
> Based on my experience with Dragon, I have pretty good handle on what the
> YAML (1) would need to look like, and how to write language-neutral helper
> code (2). What would take a little more investigation is to what extent we
> could do (3) on the basis of an understanding of the target language alone.
> E.g. if a new step or signature is added to Gremlin, will it be enough to
> add a specification of the step to the abstract grammar, or would we need
> to special-case the step for each language variant? I suspect we wouldn't
> have to do *too* much special-casing, but that's to be determined.
>
> I might suggest calling the module "gremlin-language" if we were to
> undertake the above. That would also allow other schemas to be provided
> which would help us generate the structure API into different
> implementation languages in a consistent way, as we discussed last year
> <https://www.slideshare.net/joshsh/tinkerpop-2020>.
>
Are you suggesting I change gremlin-grammar to gremlin-language with that
work in mind? I don't think it's a problem to do so. I guess
gremlin-language would eventually house (1), (2), and (3)? Right now,
though, gremlin-grammar also generates a Java parser from the one ANTLR
file I have there. Would the idea be that we'd organize this module to
generate parsers for each language we supported as well? or would that live
somewhere else?
>
> Btw. I will be giving a Category Theory and Applications presentation
> <https://www.meetup.com/Category-Theory/events/nnrhgsyccfbhc/> next week
> which will illustrate how something like the above might work.
>
> Josh
>
>
>
> On Tue, Mar 16, 2021 at 12:48 PM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > Here is the PR: https://github.com/apache/tinkerpop/pull/1408
> >
> > On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > No branch yet, but I think I will be sending the PR today.
> > >
> > > On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier <jo...@fortytwo.net>
> > > wrote:
> > >
> > >> Is there a branch we can take a look at before the PR is ready?
> > >>
> > >> Josh
> > >>
> > >> On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette <
> spmallette@gmail.com>
> > >> wrote:
> > >>
> > >> > I've been working on forming a pull request for this task. I don't
> > >> think IP
> > >> > Clearance is necessary as I originally did because the contribution
> is
> > >> > really just an ANTLR4 grammar file with some tests to validate
> things.
> > >> > Therefore, it's not a big body of independent code as I'd perhaps
> > >> initially
> > >> > envisioned. Compared to gremlint, this addition is pretty simple and
> > >> > straightforward. I've created this issue in JIRA with some
> additional
> > >> notes
> > >> > on what to expect in this initial body of work:
> > >> >
> > >> > https://issues.apache.org/jira/browse/TINKERPOP-2533
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette <
> > spmallette@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Just wanted to leave an update on this thread. It was nice to see
> > some
> > >> > > support for it. I've not had time to focus on the task itself so
> > sorry
> > >> > > there hasn't been much movement, but I hope to see it on track
> > soon. I
> > >> > > thought to update the thread after I came across yet another nice
> > >> usage
> > >> > for
> > >> > > it. I've long wanted to unify our test framework (i.e. deprecate
> the
> > >> JVM
> > >> > > process suite in favor of the GLV test suite). I was experimenting
> > >> with
> > >> > > what that might look like on Friday and hit a circular dependency
> > >> which
> > >> > > constantly trips things up where gremlin-test wants to depend on
> > >> > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy
> depends
> > >> on
> > >> > > gremlin-test and tinkergraph with <test> scope already. I think
> the
> > >> > > introduction of gremlin-script would let gremlin-test build the
> > >> Traversal
> > >> > > object from a Gremlin string and thus avoid that circular
> > >> relationship.
> > >> > >
> > >> > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail <
> > pieter.martin@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> +1
> > >> > >>
> > >> > >> I have often thought the language specification should be a
> project
> > >> > >> separate from the implementations, and done in a formal but plain
> > >> > >> English format similar to OMG or IETF specifications.
> > >> > >>
> > >> > >> I suspect Sqlg's code base would have been fastly different if it
> > had
> > >> > >> evolved from a grammer instead of an api.
> > >> > >>
> > >> > >> Cheers
> > >> > >> Pieter
> > >> > >>
> > >> > >> On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> > >> > >> > As a project, over the years, we've often been asked the
> question
> > >> as
> > >> > >> > to why
> > >> > >> > Gremlin doesn't have an ANTLR style grammar. There have been
> > >> varying
> > >> > >> > answers over the years to explain the reasoning but in recent
> > years
> > >> > >> > I've
> > >> > >> > started to see where our dependence on Java for driving Gremlin
> > >> > >> > design has
> > >> > >> > not translated well as we have expanded Gremlin into other
> > >> > >> > programming
> > >> > >> > ecosystems. Using Java has often allowed idioms of that
> language
> > to
> > >> > >> > leak
> > >> > >> > into Gremlin itself which introduces friction when implemented
> > >> > >> > outside of
> > >> > >> > the JVM. I think that there is some advantage to designing
> > Gremlin
> > >> > >> > more
> > >> > >> > with just graphs/usage in mind and then determining how that
> > design
> > >> > >> > choice
> > >> > >> > looks in each programming language.
> > >> > >> >
> > >> > >> > I think that using an ANTLR grammar to drive that design work
> for
> > >> > >> > Gremlin
> > >> > >> > makes a lot of sense in this context. We would effectively have
> > >> > >> > something
> > >> > >> > like a gremlin-script which would become the new language
> > >> archetype.
> > >> > >> > New
> > >> > >> > steps, language changes, etc. would be discussed in its context
> > and
> > >> > >> > then
> > >> > >> > implemented in the grammar and later in each programming
> language
> > >> we
> > >> > >> > support in the style a developer would expect. An interesting
> > >> upside
> > >> > >> > of
> > >> > >> > this approach is that we can implement gremlin-script in the
> > >> > >> > ScriptEngine
> > >> > >> > and replace GremlinGroovyScriptEngine which would help us
> > >> strengthen
> > >> > >> > our
> > >> > >> > security story in Gremlin Server. Groovy processing would just
> > be a
> > >> > >> > fallback to Gremlin scripts that could not be processed by the
> > AST.
> > >> > >> > In fact
> > >> > >> > users who didn't need Groovy could simply not install it at all
> > and
> > >> > >> > thus
> > >> > >> > boast a much more secure system.
> > >> > >> >
> > >> > >> > I think that inclusion of a grammar in our project is an
> exciting
> > >> new
> > >> > >> > direction for us to take and will help in a variety of areas
> > beyond
> > >> > >> > those
> > >> > >> > I've already related.
> > >> > >> >
> > >> > >> > If we like this direction, Amazon Neptune already maintains
> such
> > a
> > >> > >> > grammar
> > >> > >> > and would be willing to contribute it to the project to live in
> > >> open
> > >> > >> > source. The contribution would go through the same IP Clearance
> > >> > >> > process
> > >> > >> > gremlint is going through since it was developed outside of
> > >> > >> > TinkerPop. I'd
> > >> > >> > be happy to guide that process through if we draw to consensus
> > >> here.
> > >> > >>
> > >> > >>
> > >> > >>
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Joshua Shinavier <jo...@fortytwo.net>.
Awesome. This definitely fills a major gap, and as Florian said, it is
great that it is already improving the documentation.
High level-question: do you have any thoughts about Gremlin syntax
validation *in general*, i.e. for language variants other than Groovy? It
would be interesting to explore the following:
1. Specify an abstract Gremlin grammar in a neutral language like YAML
2. Write some helper code for generating ANTLR grammars from the YAML
3. For each Gremlin language variant, write a smaller amount of code
based on (2) to generate a language-specific ANTLR grammar
Based on my experience with Dragon, I have pretty good handle on what the
YAML (1) would need to look like, and how to write language-neutral helper
code (2). What would take a little more investigation is to what extent we
could do (3) on the basis of an understanding of the target language alone.
E.g. if a new step or signature is added to Gremlin, will it be enough to
add a specification of the step to the abstract grammar, or would we need
to special-case the step for each language variant? I suspect we wouldn't
have to do *too* much special-casing, but that's to be determined.
I might suggest calling the module "gremlin-language" if we were to
undertake the above. That would also allow other schemas to be provided
which would help us generate the structure API into different
implementation languages in a consistent way, as we discussed last year
<https://www.slideshare.net/joshsh/tinkerpop-2020>.
Btw. I will be giving a Category Theory and Applications presentation
<https://www.meetup.com/Category-Theory/events/nnrhgsyccfbhc/> next week
which will illustrate how something like the above might work.
Josh
On Tue, Mar 16, 2021 at 12:48 PM Stephen Mallette <sp...@gmail.com>
wrote:
> Here is the PR: https://github.com/apache/tinkerpop/pull/1408
>
> On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > No branch yet, but I think I will be sending the PR today.
> >
> > On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier <jo...@fortytwo.net>
> > wrote:
> >
> >> Is there a branch we can take a look at before the PR is ready?
> >>
> >> Josh
> >>
> >> On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette <sp...@gmail.com>
> >> wrote:
> >>
> >> > I've been working on forming a pull request for this task. I don't
> >> think IP
> >> > Clearance is necessary as I originally did because the contribution is
> >> > really just an ANTLR4 grammar file with some tests to validate things.
> >> > Therefore, it's not a big body of independent code as I'd perhaps
> >> initially
> >> > envisioned. Compared to gremlint, this addition is pretty simple and
> >> > straightforward. I've created this issue in JIRA with some additional
> >> notes
> >> > on what to expect in this initial body of work:
> >> >
> >> > https://issues.apache.org/jira/browse/TINKERPOP-2533
> >> >
> >> >
> >> >
> >> > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette <
> spmallette@gmail.com>
> >> > wrote:
> >> >
> >> > > Just wanted to leave an update on this thread. It was nice to see
> some
> >> > > support for it. I've not had time to focus on the task itself so
> sorry
> >> > > there hasn't been much movement, but I hope to see it on track
> soon. I
> >> > > thought to update the thread after I came across yet another nice
> >> usage
> >> > for
> >> > > it. I've long wanted to unify our test framework (i.e. deprecate the
> >> JVM
> >> > > process suite in favor of the GLV test suite). I was experimenting
> >> with
> >> > > what that might look like on Friday and hit a circular dependency
> >> which
> >> > > constantly trips things up where gremlin-test wants to depend on
> >> > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy depends
> >> on
> >> > > gremlin-test and tinkergraph with <test> scope already. I think the
> >> > > introduction of gremlin-script would let gremlin-test build the
> >> Traversal
> >> > > object from a Gremlin string and thus avoid that circular
> >> relationship.
> >> > >
> >> > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail <
> pieter.martin@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> +1
> >> > >>
> >> > >> I have often thought the language specification should be a project
> >> > >> separate from the implementations, and done in a formal but plain
> >> > >> English format similar to OMG or IETF specifications.
> >> > >>
> >> > >> I suspect Sqlg's code base would have been fastly different if it
> had
> >> > >> evolved from a grammer instead of an api.
> >> > >>
> >> > >> Cheers
> >> > >> Pieter
> >> > >>
> >> > >> On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> >> > >> > As a project, over the years, we've often been asked the question
> >> as
> >> > >> > to why
> >> > >> > Gremlin doesn't have an ANTLR style grammar. There have been
> >> varying
> >> > >> > answers over the years to explain the reasoning but in recent
> years
> >> > >> > I've
> >> > >> > started to see where our dependence on Java for driving Gremlin
> >> > >> > design has
> >> > >> > not translated well as we have expanded Gremlin into other
> >> > >> > programming
> >> > >> > ecosystems. Using Java has often allowed idioms of that language
> to
> >> > >> > leak
> >> > >> > into Gremlin itself which introduces friction when implemented
> >> > >> > outside of
> >> > >> > the JVM. I think that there is some advantage to designing
> Gremlin
> >> > >> > more
> >> > >> > with just graphs/usage in mind and then determining how that
> design
> >> > >> > choice
> >> > >> > looks in each programming language.
> >> > >> >
> >> > >> > I think that using an ANTLR grammar to drive that design work for
> >> > >> > Gremlin
> >> > >> > makes a lot of sense in this context. We would effectively have
> >> > >> > something
> >> > >> > like a gremlin-script which would become the new language
> >> archetype.
> >> > >> > New
> >> > >> > steps, language changes, etc. would be discussed in its context
> and
> >> > >> > then
> >> > >> > implemented in the grammar and later in each programming language
> >> we
> >> > >> > support in the style a developer would expect. An interesting
> >> upside
> >> > >> > of
> >> > >> > this approach is that we can implement gremlin-script in the
> >> > >> > ScriptEngine
> >> > >> > and replace GremlinGroovyScriptEngine which would help us
> >> strengthen
> >> > >> > our
> >> > >> > security story in Gremlin Server. Groovy processing would just
> be a
> >> > >> > fallback to Gremlin scripts that could not be processed by the
> AST.
> >> > >> > In fact
> >> > >> > users who didn't need Groovy could simply not install it at all
> and
> >> > >> > thus
> >> > >> > boast a much more secure system.
> >> > >> >
> >> > >> > I think that inclusion of a grammar in our project is an exciting
> >> new
> >> > >> > direction for us to take and will help in a variety of areas
> beyond
> >> > >> > those
> >> > >> > I've already related.
> >> > >> >
> >> > >> > If we like this direction, Amazon Neptune already maintains such
> a
> >> > >> > grammar
> >> > >> > and would be willing to contribute it to the project to live in
> >> open
> >> > >> > source. The contribution would go through the same IP Clearance
> >> > >> > process
> >> > >> > gremlint is going through since it was developed outside of
> >> > >> > TinkerPop. I'd
> >> > >> > be happy to guide that process through if we draw to consensus
> >> here.
> >> > >>
> >> > >>
> >> > >>
> >> >
> >>
> >
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Stephen Mallette <sp...@gmail.com>.
Here is the PR: https://github.com/apache/tinkerpop/pull/1408
On Tue, Mar 16, 2021 at 6:14 AM Stephen Mallette <sp...@gmail.com>
wrote:
> No branch yet, but I think I will be sending the PR today.
>
> On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier <jo...@fortytwo.net>
> wrote:
>
>> Is there a branch we can take a look at before the PR is ready?
>>
>> Josh
>>
>> On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette <sp...@gmail.com>
>> wrote:
>>
>> > I've been working on forming a pull request for this task. I don't
>> think IP
>> > Clearance is necessary as I originally did because the contribution is
>> > really just an ANTLR4 grammar file with some tests to validate things.
>> > Therefore, it's not a big body of independent code as I'd perhaps
>> initially
>> > envisioned. Compared to gremlint, this addition is pretty simple and
>> > straightforward. I've created this issue in JIRA with some additional
>> notes
>> > on what to expect in this initial body of work:
>> >
>> > https://issues.apache.org/jira/browse/TINKERPOP-2533
>> >
>> >
>> >
>> > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette <sp...@gmail.com>
>> > wrote:
>> >
>> > > Just wanted to leave an update on this thread. It was nice to see some
>> > > support for it. I've not had time to focus on the task itself so sorry
>> > > there hasn't been much movement, but I hope to see it on track soon. I
>> > > thought to update the thread after I came across yet another nice
>> usage
>> > for
>> > > it. I've long wanted to unify our test framework (i.e. deprecate the
>> JVM
>> > > process suite in favor of the GLV test suite). I was experimenting
>> with
>> > > what that might look like on Friday and hit a circular dependency
>> which
>> > > constantly trips things up where gremlin-test wants to depend on
>> > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy depends
>> on
>> > > gremlin-test and tinkergraph with <test> scope already. I think the
>> > > introduction of gremlin-script would let gremlin-test build the
>> Traversal
>> > > object from a Gremlin string and thus avoid that circular
>> relationship.
>> > >
>> > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail <pi...@gmail.com>
>> > > wrote:
>> > >
>> > >> +1
>> > >>
>> > >> I have often thought the language specification should be a project
>> > >> separate from the implementations, and done in a formal but plain
>> > >> English format similar to OMG or IETF specifications.
>> > >>
>> > >> I suspect Sqlg's code base would have been fastly different if it had
>> > >> evolved from a grammer instead of an api.
>> > >>
>> > >> Cheers
>> > >> Pieter
>> > >>
>> > >> On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
>> > >> > As a project, over the years, we've often been asked the question
>> as
>> > >> > to why
>> > >> > Gremlin doesn't have an ANTLR style grammar. There have been
>> varying
>> > >> > answers over the years to explain the reasoning but in recent years
>> > >> > I've
>> > >> > started to see where our dependence on Java for driving Gremlin
>> > >> > design has
>> > >> > not translated well as we have expanded Gremlin into other
>> > >> > programming
>> > >> > ecosystems. Using Java has often allowed idioms of that language to
>> > >> > leak
>> > >> > into Gremlin itself which introduces friction when implemented
>> > >> > outside of
>> > >> > the JVM. I think that there is some advantage to designing Gremlin
>> > >> > more
>> > >> > with just graphs/usage in mind and then determining how that design
>> > >> > choice
>> > >> > looks in each programming language.
>> > >> >
>> > >> > I think that using an ANTLR grammar to drive that design work for
>> > >> > Gremlin
>> > >> > makes a lot of sense in this context. We would effectively have
>> > >> > something
>> > >> > like a gremlin-script which would become the new language
>> archetype.
>> > >> > New
>> > >> > steps, language changes, etc. would be discussed in its context and
>> > >> > then
>> > >> > implemented in the grammar and later in each programming language
>> we
>> > >> > support in the style a developer would expect. An interesting
>> upside
>> > >> > of
>> > >> > this approach is that we can implement gremlin-script in the
>> > >> > ScriptEngine
>> > >> > and replace GremlinGroovyScriptEngine which would help us
>> strengthen
>> > >> > our
>> > >> > security story in Gremlin Server. Groovy processing would just be a
>> > >> > fallback to Gremlin scripts that could not be processed by the AST.
>> > >> > In fact
>> > >> > users who didn't need Groovy could simply not install it at all and
>> > >> > thus
>> > >> > boast a much more secure system.
>> > >> >
>> > >> > I think that inclusion of a grammar in our project is an exciting
>> new
>> > >> > direction for us to take and will help in a variety of areas beyond
>> > >> > those
>> > >> > I've already related.
>> > >> >
>> > >> > If we like this direction, Amazon Neptune already maintains such a
>> > >> > grammar
>> > >> > and would be willing to contribute it to the project to live in
>> open
>> > >> > source. The contribution would go through the same IP Clearance
>> > >> > process
>> > >> > gremlint is going through since it was developed outside of
>> > >> > TinkerPop. I'd
>> > >> > be happy to guide that process through if we draw to consensus
>> here.
>> > >>
>> > >>
>> > >>
>> >
>>
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Stephen Mallette <sp...@gmail.com>.
No branch yet, but I think I will be sending the PR today.
On Mon, Mar 15, 2021 at 9:33 PM Joshua Shinavier <jo...@fortytwo.net> wrote:
> Is there a branch we can take a look at before the PR is ready?
>
> Josh
>
> On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > I've been working on forming a pull request for this task. I don't think
> IP
> > Clearance is necessary as I originally did because the contribution is
> > really just an ANTLR4 grammar file with some tests to validate things.
> > Therefore, it's not a big body of independent code as I'd perhaps
> initially
> > envisioned. Compared to gremlint, this addition is pretty simple and
> > straightforward. I've created this issue in JIRA with some additional
> notes
> > on what to expect in this initial body of work:
> >
> > https://issues.apache.org/jira/browse/TINKERPOP-2533
> >
> >
> >
> > On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > Just wanted to leave an update on this thread. It was nice to see some
> > > support for it. I've not had time to focus on the task itself so sorry
> > > there hasn't been much movement, but I hope to see it on track soon. I
> > > thought to update the thread after I came across yet another nice usage
> > for
> > > it. I've long wanted to unify our test framework (i.e. deprecate the
> JVM
> > > process suite in favor of the GLV test suite). I was experimenting with
> > > what that might look like on Friday and hit a circular dependency which
> > > constantly trips things up where gremlin-test wants to depend on
> > > gremlin-groovy (for ScriptEngine support) but gremlin-groovy depends on
> > > gremlin-test and tinkergraph with <test> scope already. I think the
> > > introduction of gremlin-script would let gremlin-test build the
> Traversal
> > > object from a Gremlin string and thus avoid that circular relationship.
> > >
> > > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail <pi...@gmail.com>
> > > wrote:
> > >
> > >> +1
> > >>
> > >> I have often thought the language specification should be a project
> > >> separate from the implementations, and done in a formal but plain
> > >> English format similar to OMG or IETF specifications.
> > >>
> > >> I suspect Sqlg's code base would have been fastly different if it had
> > >> evolved from a grammer instead of an api.
> > >>
> > >> Cheers
> > >> Pieter
> > >>
> > >> On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> > >> > As a project, over the years, we've often been asked the question as
> > >> > to why
> > >> > Gremlin doesn't have an ANTLR style grammar. There have been varying
> > >> > answers over the years to explain the reasoning but in recent years
> > >> > I've
> > >> > started to see where our dependence on Java for driving Gremlin
> > >> > design has
> > >> > not translated well as we have expanded Gremlin into other
> > >> > programming
> > >> > ecosystems. Using Java has often allowed idioms of that language to
> > >> > leak
> > >> > into Gremlin itself which introduces friction when implemented
> > >> > outside of
> > >> > the JVM. I think that there is some advantage to designing Gremlin
> > >> > more
> > >> > with just graphs/usage in mind and then determining how that design
> > >> > choice
> > >> > looks in each programming language.
> > >> >
> > >> > I think that using an ANTLR grammar to drive that design work for
> > >> > Gremlin
> > >> > makes a lot of sense in this context. We would effectively have
> > >> > something
> > >> > like a gremlin-script which would become the new language archetype.
> > >> > New
> > >> > steps, language changes, etc. would be discussed in its context and
> > >> > then
> > >> > implemented in the grammar and later in each programming language we
> > >> > support in the style a developer would expect. An interesting upside
> > >> > of
> > >> > this approach is that we can implement gremlin-script in the
> > >> > ScriptEngine
> > >> > and replace GremlinGroovyScriptEngine which would help us strengthen
> > >> > our
> > >> > security story in Gremlin Server. Groovy processing would just be a
> > >> > fallback to Gremlin scripts that could not be processed by the AST.
> > >> > In fact
> > >> > users who didn't need Groovy could simply not install it at all and
> > >> > thus
> > >> > boast a much more secure system.
> > >> >
> > >> > I think that inclusion of a grammar in our project is an exciting
> new
> > >> > direction for us to take and will help in a variety of areas beyond
> > >> > those
> > >> > I've already related.
> > >> >
> > >> > If we like this direction, Amazon Neptune already maintains such a
> > >> > grammar
> > >> > and would be willing to contribute it to the project to live in open
> > >> > source. The contribution would go through the same IP Clearance
> > >> > process
> > >> > gremlint is going through since it was developed outside of
> > >> > TinkerPop. I'd
> > >> > be happy to guide that process through if we draw to consensus here.
> > >>
> > >>
> > >>
> >
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Joshua Shinavier <jo...@fortytwo.net>.
Is there a branch we can take a look at before the PR is ready?
Josh
On Fri, Mar 12, 2021 at 5:42 AM Stephen Mallette <sp...@gmail.com>
wrote:
> I've been working on forming a pull request for this task. I don't think IP
> Clearance is necessary as I originally did because the contribution is
> really just an ANTLR4 grammar file with some tests to validate things.
> Therefore, it's not a big body of independent code as I'd perhaps initially
> envisioned. Compared to gremlint, this addition is pretty simple and
> straightforward. I've created this issue in JIRA with some additional notes
> on what to expect in this initial body of work:
>
> https://issues.apache.org/jira/browse/TINKERPOP-2533
>
>
>
> On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > Just wanted to leave an update on this thread. It was nice to see some
> > support for it. I've not had time to focus on the task itself so sorry
> > there hasn't been much movement, but I hope to see it on track soon. I
> > thought to update the thread after I came across yet another nice usage
> for
> > it. I've long wanted to unify our test framework (i.e. deprecate the JVM
> > process suite in favor of the GLV test suite). I was experimenting with
> > what that might look like on Friday and hit a circular dependency which
> > constantly trips things up where gremlin-test wants to depend on
> > gremlin-groovy (for ScriptEngine support) but gremlin-groovy depends on
> > gremlin-test and tinkergraph with <test> scope already. I think the
> > introduction of gremlin-script would let gremlin-test build the Traversal
> > object from a Gremlin string and thus avoid that circular relationship.
> >
> > On Fri, Jan 8, 2021 at 2:43 AM pieter gmail <pi...@gmail.com>
> > wrote:
> >
> >> +1
> >>
> >> I have often thought the language specification should be a project
> >> separate from the implementations, and done in a formal but plain
> >> English format similar to OMG or IETF specifications.
> >>
> >> I suspect Sqlg's code base would have been fastly different if it had
> >> evolved from a grammer instead of an api.
> >>
> >> Cheers
> >> Pieter
> >>
> >> On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> >> > As a project, over the years, we've often been asked the question as
> >> > to why
> >> > Gremlin doesn't have an ANTLR style grammar. There have been varying
> >> > answers over the years to explain the reasoning but in recent years
> >> > I've
> >> > started to see where our dependence on Java for driving Gremlin
> >> > design has
> >> > not translated well as we have expanded Gremlin into other
> >> > programming
> >> > ecosystems. Using Java has often allowed idioms of that language to
> >> > leak
> >> > into Gremlin itself which introduces friction when implemented
> >> > outside of
> >> > the JVM. I think that there is some advantage to designing Gremlin
> >> > more
> >> > with just graphs/usage in mind and then determining how that design
> >> > choice
> >> > looks in each programming language.
> >> >
> >> > I think that using an ANTLR grammar to drive that design work for
> >> > Gremlin
> >> > makes a lot of sense in this context. We would effectively have
> >> > something
> >> > like a gremlin-script which would become the new language archetype.
> >> > New
> >> > steps, language changes, etc. would be discussed in its context and
> >> > then
> >> > implemented in the grammar and later in each programming language we
> >> > support in the style a developer would expect. An interesting upside
> >> > of
> >> > this approach is that we can implement gremlin-script in the
> >> > ScriptEngine
> >> > and replace GremlinGroovyScriptEngine which would help us strengthen
> >> > our
> >> > security story in Gremlin Server. Groovy processing would just be a
> >> > fallback to Gremlin scripts that could not be processed by the AST.
> >> > In fact
> >> > users who didn't need Groovy could simply not install it at all and
> >> > thus
> >> > boast a much more secure system.
> >> >
> >> > I think that inclusion of a grammar in our project is an exciting new
> >> > direction for us to take and will help in a variety of areas beyond
> >> > those
> >> > I've already related.
> >> >
> >> > If we like this direction, Amazon Neptune already maintains such a
> >> > grammar
> >> > and would be willing to contribute it to the project to live in open
> >> > source. The contribution would go through the same IP Clearance
> >> > process
> >> > gremlint is going through since it was developed outside of
> >> > TinkerPop. I'd
> >> > be happy to guide that process through if we draw to consensus here.
> >>
> >>
> >>
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Stephen Mallette <sp...@gmail.com>.
I've been working on forming a pull request for this task. I don't think IP
Clearance is necessary as I originally did because the contribution is
really just an ANTLR4 grammar file with some tests to validate things.
Therefore, it's not a big body of independent code as I'd perhaps initially
envisioned. Compared to gremlint, this addition is pretty simple and
straightforward. I've created this issue in JIRA with some additional notes
on what to expect in this initial body of work:
https://issues.apache.org/jira/browse/TINKERPOP-2533
On Mon, Feb 8, 2021 at 10:06 AM Stephen Mallette <sp...@gmail.com>
wrote:
> Just wanted to leave an update on this thread. It was nice to see some
> support for it. I've not had time to focus on the task itself so sorry
> there hasn't been much movement, but I hope to see it on track soon. I
> thought to update the thread after I came across yet another nice usage for
> it. I've long wanted to unify our test framework (i.e. deprecate the JVM
> process suite in favor of the GLV test suite). I was experimenting with
> what that might look like on Friday and hit a circular dependency which
> constantly trips things up where gremlin-test wants to depend on
> gremlin-groovy (for ScriptEngine support) but gremlin-groovy depends on
> gremlin-test and tinkergraph with <test> scope already. I think the
> introduction of gremlin-script would let gremlin-test build the Traversal
> object from a Gremlin string and thus avoid that circular relationship.
>
> On Fri, Jan 8, 2021 at 2:43 AM pieter gmail <pi...@gmail.com>
> wrote:
>
>> +1
>>
>> I have often thought the language specification should be a project
>> separate from the implementations, and done in a formal but plain
>> English format similar to OMG or IETF specifications.
>>
>> I suspect Sqlg's code base would have been fastly different if it had
>> evolved from a grammer instead of an api.
>>
>> Cheers
>> Pieter
>>
>> On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
>> > As a project, over the years, we've often been asked the question as
>> > to why
>> > Gremlin doesn't have an ANTLR style grammar. There have been varying
>> > answers over the years to explain the reasoning but in recent years
>> > I've
>> > started to see where our dependence on Java for driving Gremlin
>> > design has
>> > not translated well as we have expanded Gremlin into other
>> > programming
>> > ecosystems. Using Java has often allowed idioms of that language to
>> > leak
>> > into Gremlin itself which introduces friction when implemented
>> > outside of
>> > the JVM. I think that there is some advantage to designing Gremlin
>> > more
>> > with just graphs/usage in mind and then determining how that design
>> > choice
>> > looks in each programming language.
>> >
>> > I think that using an ANTLR grammar to drive that design work for
>> > Gremlin
>> > makes a lot of sense in this context. We would effectively have
>> > something
>> > like a gremlin-script which would become the new language archetype.
>> > New
>> > steps, language changes, etc. would be discussed in its context and
>> > then
>> > implemented in the grammar and later in each programming language we
>> > support in the style a developer would expect. An interesting upside
>> > of
>> > this approach is that we can implement gremlin-script in the
>> > ScriptEngine
>> > and replace GremlinGroovyScriptEngine which would help us strengthen
>> > our
>> > security story in Gremlin Server. Groovy processing would just be a
>> > fallback to Gremlin scripts that could not be processed by the AST.
>> > In fact
>> > users who didn't need Groovy could simply not install it at all and
>> > thus
>> > boast a much more secure system.
>> >
>> > I think that inclusion of a grammar in our project is an exciting new
>> > direction for us to take and will help in a variety of areas beyond
>> > those
>> > I've already related.
>> >
>> > If we like this direction, Amazon Neptune already maintains such a
>> > grammar
>> > and would be willing to contribute it to the project to live in open
>> > source. The contribution would go through the same IP Clearance
>> > process
>> > gremlint is going through since it was developed outside of
>> > TinkerPop. I'd
>> > be happy to guide that process through if we draw to consensus here.
>>
>>
>>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by Stephen Mallette <sp...@gmail.com>.
Just wanted to leave an update on this thread. It was nice to see some
support for it. I've not had time to focus on the task itself so sorry
there hasn't been much movement, but I hope to see it on track soon. I
thought to update the thread after I came across yet another nice usage for
it. I've long wanted to unify our test framework (i.e. deprecate the JVM
process suite in favor of the GLV test suite). I was experimenting with
what that might look like on Friday and hit a circular dependency which
constantly trips things up where gremlin-test wants to depend on
gremlin-groovy (for ScriptEngine support) but gremlin-groovy depends on
gremlin-test and tinkergraph with <test> scope already. I think the
introduction of gremlin-script would let gremlin-test build the Traversal
object from a Gremlin string and thus avoid that circular relationship.
On Fri, Jan 8, 2021 at 2:43 AM pieter gmail <pi...@gmail.com> wrote:
> +1
>
> I have often thought the language specification should be a project
> separate from the implementations, and done in a formal but plain
> English format similar to OMG or IETF specifications.
>
> I suspect Sqlg's code base would have been fastly different if it had
> evolved from a grammer instead of an api.
>
> Cheers
> Pieter
>
> On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> > As a project, over the years, we've often been asked the question as
> > to why
> > Gremlin doesn't have an ANTLR style grammar. There have been varying
> > answers over the years to explain the reasoning but in recent years
> > I've
> > started to see where our dependence on Java for driving Gremlin
> > design has
> > not translated well as we have expanded Gremlin into other
> > programming
> > ecosystems. Using Java has often allowed idioms of that language to
> > leak
> > into Gremlin itself which introduces friction when implemented
> > outside of
> > the JVM. I think that there is some advantage to designing Gremlin
> > more
> > with just graphs/usage in mind and then determining how that design
> > choice
> > looks in each programming language.
> >
> > I think that using an ANTLR grammar to drive that design work for
> > Gremlin
> > makes a lot of sense in this context. We would effectively have
> > something
> > like a gremlin-script which would become the new language archetype.
> > New
> > steps, language changes, etc. would be discussed in its context and
> > then
> > implemented in the grammar and later in each programming language we
> > support in the style a developer would expect. An interesting upside
> > of
> > this approach is that we can implement gremlin-script in the
> > ScriptEngine
> > and replace GremlinGroovyScriptEngine which would help us strengthen
> > our
> > security story in Gremlin Server. Groovy processing would just be a
> > fallback to Gremlin scripts that could not be processed by the AST.
> > In fact
> > users who didn't need Groovy could simply not install it at all and
> > thus
> > boast a much more secure system.
> >
> > I think that inclusion of a grammar in our project is an exciting new
> > direction for us to take and will help in a variety of areas beyond
> > those
> > I've already related.
> >
> > If we like this direction, Amazon Neptune already maintains such a
> > grammar
> > and would be willing to contribute it to the project to live in open
> > source. The contribution would go through the same IP Clearance
> > process
> > gremlint is going through since it was developed outside of
> > TinkerPop. I'd
> > be happy to guide that process through if we draw to consensus here.
>
>
>
Re: [DISCUSS] ANTLR and gremlin-script
Posted by pieter gmail <pi...@gmail.com>.
+1
I have often thought the language specification should be a project
separate from the implementations, and done in a formal but plain
English format similar to OMG or IETF specifications.
I suspect Sqlg's code base would have been fastly different if it had
evolved from a grammer instead of an api.
Cheers
Pieter
On Thu, 2020-12-24 at 14:41 -0500, Stephen Mallette wrote:
> As a project, over the years, we've often been asked the question as
> to why
> Gremlin doesn't have an ANTLR style grammar. There have been varying
> answers over the years to explain the reasoning but in recent years
> I've
> started to see where our dependence on Java for driving Gremlin
> design has
> not translated well as we have expanded Gremlin into other
> programming
> ecosystems. Using Java has often allowed idioms of that language to
> leak
> into Gremlin itself which introduces friction when implemented
> outside of
> the JVM. I think that there is some advantage to designing Gremlin
> more
> with just graphs/usage in mind and then determining how that design
> choice
> looks in each programming language.
>
> I think that using an ANTLR grammar to drive that design work for
> Gremlin
> makes a lot of sense in this context. We would effectively have
> something
> like a gremlin-script which would become the new language archetype.
> New
> steps, language changes, etc. would be discussed in its context and
> then
> implemented in the grammar and later in each programming language we
> support in the style a developer would expect. An interesting upside
> of
> this approach is that we can implement gremlin-script in the
> ScriptEngine
> and replace GremlinGroovyScriptEngine which would help us strengthen
> our
> security story in Gremlin Server. Groovy processing would just be a
> fallback to Gremlin scripts that could not be processed by the AST.
> In fact
> users who didn't need Groovy could simply not install it at all and
> thus
> boast a much more secure system.
>
> I think that inclusion of a grammar in our project is an exciting new
> direction for us to take and will help in a variety of areas beyond
> those
> I've already related.
>
> If we like this direction, Amazon Neptune already maintains such a
> grammar
> and would be willing to contribute it to the project to live in open
> source. The contribution would go through the same IP Clearance
> process
> gremlint is going through since it was developed outside of
> TinkerPop. I'd
> be happy to guide that process through if we draw to consensus here.