You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@groovy.apache.org by Jesper Steen Møller <je...@selskabet.org> on 2016/02/24 01:14:01 UTC
Antlr4-based Groovy parser status update
Hi Groovy-dev
I’ve found some time to work on the Antlr4 parser and fixed some minor tasks there:
* Fix grammar for several statement sequences, where NL or semicolon was not required (added negative test cases)
* Remove non-reentrant static state in ASTBuilder
* Added syntax error handling to Antlr 4 parser
* Make sure GrammarPredicates work with arbitrarily nested package hierarchies
I’m pushing my changes here:
https://github.com/jespersm/groovy/commits/antlr4
Next up is support for non-ID method invocation and command expressions, and then unary expressions.
Also, since the GSoC is a bit dated, traits are also missing.
-Jesper
Re: Antlr4-based Groovy parser status update
Posted by Paul King <pa...@asert.com.au>.
Ah yes, I'd forgotten about that line of thinking. I still believe we
should be able to set things up so that prod code isn't affected by
Spock but I don't have time to investigate further right now so being
extra cautious as per Cédric's comment is the way to go.
Cheers, Paul.
On Wed, Feb 24, 2016 at 10:24 PM, Jesper Steen Møller
<je...@selskabet.org> wrote:
> Ah, good to know.
>
> While merging in the master changes since the antlr4 fork, I noticed that
> Spock was indeed in use already, so I haven’t ditched it — yet. For now,
> it’s only used for simple things like where-blocks, which can be done only
> slightly more clumsily with JUnit.
>
> -Jesper
>
> On 24. feb. 2016, at 12.41, Cédric Champeau <ce...@gmail.com>
> wrote:
>
> It is a show stopper for groovy-core. We must *not* introduce a dependency
> on Spock, because it would conflict with the version of Groovy that we are
> using, and apply global xforms on Groovy core. While it might be ok, we want
> to minimize the risks.
>
> 2016-02-24 12:35 GMT+01:00 Paul King <pa...@asert.com.au>:
>>
>> Spock for testing is by no means a show stopper!
>>
>> On Wed, Feb 24, 2016 at 6:46 PM, Jesper Steen Møller
>> <je...@selskabet.org> wrote:
>> >
>> > On 24. feb. 2016, at 01.21, Guillaume Laforge <gl...@gmail.com>
>> > wrote:
>> >
>> > Sounds cool, Jesper, great to hear about your progress on the Antlr4
>> > grammar!
>> > Is there a way we can help you? How can we test drive this? Is the build
>> > working?
>> >
>> >
>> > The build works in that the antlr4 branch can be build and tested with
>> > e.g.
>> > “gradle -PuseAntlr4=true test”.
>> >
>> > However, I can’t seem to actually build a real distribution and use the
>> > new
>> > parser in that (there’s a dynamic switch), without getting some very
>> > strange
>> > errors.
>> > I’ve put off chasing that problem until I know I have the missing pieces
>> > of
>> > the grammar and the AST building done right.
>> >
>> > The current tests use Spock. Is that out of the question for the main
>> > Groovy
>> > build? I loooove Spock, but I realise we have to keep the number of
>> > dependencies down.
>> >
>> > -Jesper
>> >
>> >
>> > On Wed, Feb 24, 2016 at 1:14 AM, Jesper Steen Møller
>> > <je...@selskabet.org>
>> > wrote:
>> >>
>> >> Hi Groovy-dev
>> >>
>> >> I’ve found some time to work on the Antlr4 parser and fixed some minor
>> >> tasks there:
>> >>
>> >> * Fix grammar for several statement sequences, where NL or semicolon
>> >> was
>> >> not required (added negative test cases)
>> >> * Remove non-reentrant static state in ASTBuilder
>> >> * Added syntax error handling to Antlr 4 parser
>> >> * Make sure GrammarPredicates work with arbitrarily nested package
>> >> hierarchies
>> >>
>> >> I’m pushing my changes here:
>> >> https://github.com/jespersm/groovy/commits/antlr4
>> >>
>> >> Next up is support for non-ID method invocation and command
>> >> expressions,
>> >> and then unary expressions.
>> >> Also, since the GSoC is a bit dated, traits are also missing.
>> >>
>> >> -Jesper
>> >
>> >
>> >
>> >
>> > --
>> > Guillaume Laforge
>> > Apache Groovy committer & PMC Vice-President
>> > Product Ninja & Advocate at Restlet
>> >
>> > Blog: http://glaforge.appspot.com/
>> > Social: @glaforge / Google+
>> >
>> >
>
>
>
Re: Antlr4-based Groovy parser status update
Posted by Jesper Steen Møller <je...@selskabet.org>.
Ah, good to know.
While merging in the master changes since the antlr4 fork, I noticed that Spock was indeed in use already, so I haven’t ditched it — yet. For now, it’s only used for simple things like where-blocks, which can be done only slightly more clumsily with JUnit.
-Jesper
> On 24. feb. 2016, at 12.41, Cédric Champeau <ce...@gmail.com> wrote:
>
> It is a show stopper for groovy-core. We must *not* introduce a dependency on Spock, because it would conflict with the version of Groovy that we are using, and apply global xforms on Groovy core. While it might be ok, we want to minimize the risks.
>
> 2016-02-24 12:35 GMT+01:00 Paul King <paulk@asert.com.au <ma...@asert.com.au>>:
> Spock for testing is by no means a show stopper!
>
> On Wed, Feb 24, 2016 at 6:46 PM, Jesper Steen Møller
> <jesper@selskabet.org <ma...@selskabet.org>> wrote:
> >
> > On 24. feb. 2016, at 01.21, Guillaume Laforge <glaforge@gmail.com <ma...@gmail.com>> wrote:
> >
> > Sounds cool, Jesper, great to hear about your progress on the Antlr4
> > grammar!
> > Is there a way we can help you? How can we test drive this? Is the build
> > working?
> >
> >
> > The build works in that the antlr4 branch can be build and tested with e.g.
> > “gradle -PuseAntlr4=true test”.
> >
> > However, I can’t seem to actually build a real distribution and use the new
> > parser in that (there’s a dynamic switch), without getting some very strange
> > errors.
> > I’ve put off chasing that problem until I know I have the missing pieces of
> > the grammar and the AST building done right.
> >
> > The current tests use Spock. Is that out of the question for the main Groovy
> > build? I loooove Spock, but I realise we have to keep the number of
> > dependencies down.
> >
> > -Jesper
> >
> >
> > On Wed, Feb 24, 2016 at 1:14 AM, Jesper Steen Møller <jesper@selskabet.org <ma...@selskabet.org>>
> > wrote:
> >>
> >> Hi Groovy-dev
> >>
> >> I’ve found some time to work on the Antlr4 parser and fixed some minor
> >> tasks there:
> >>
> >> * Fix grammar for several statement sequences, where NL or semicolon was
> >> not required (added negative test cases)
> >> * Remove non-reentrant static state in ASTBuilder
> >> * Added syntax error handling to Antlr 4 parser
> >> * Make sure GrammarPredicates work with arbitrarily nested package
> >> hierarchies
> >>
> >> I’m pushing my changes here:
> >> https://github.com/jespersm/groovy/commits/antlr4 <https://github.com/jespersm/groovy/commits/antlr4>
> >>
> >> Next up is support for non-ID method invocation and command expressions,
> >> and then unary expressions.
> >> Also, since the GSoC is a bit dated, traits are also missing.
> >>
> >> -Jesper
> >
> >
> >
> >
> > --
> > Guillaume Laforge
> > Apache Groovy committer & PMC Vice-President
> > Product Ninja & Advocate at Restlet
> >
> > Blog: http://glaforge.appspot.com/ <http://glaforge.appspot.com/>
> > Social: @glaforge / Google+
> >
> >
>
Re: Antlr4-based Groovy parser status update
Posted by Cédric Champeau <ce...@gmail.com>.
It is a show stopper for groovy-core. We must *not* introduce a dependency
on Spock, because it would conflict with the version of Groovy that we are
using, and apply global xforms on Groovy core. While it might be ok, we
want to minimize the risks.
2016-02-24 12:35 GMT+01:00 Paul King <pa...@asert.com.au>:
> Spock for testing is by no means a show stopper!
>
> On Wed, Feb 24, 2016 at 6:46 PM, Jesper Steen Møller
> <je...@selskabet.org> wrote:
> >
> > On 24. feb. 2016, at 01.21, Guillaume Laforge <gl...@gmail.com>
> wrote:
> >
> > Sounds cool, Jesper, great to hear about your progress on the Antlr4
> > grammar!
> > Is there a way we can help you? How can we test drive this? Is the build
> > working?
> >
> >
> > The build works in that the antlr4 branch can be build and tested with
> e.g.
> > “gradle -PuseAntlr4=true test”.
> >
> > However, I can’t seem to actually build a real distribution and use the
> new
> > parser in that (there’s a dynamic switch), without getting some very
> strange
> > errors.
> > I’ve put off chasing that problem until I know I have the missing pieces
> of
> > the grammar and the AST building done right.
> >
> > The current tests use Spock. Is that out of the question for the main
> Groovy
> > build? I loooove Spock, but I realise we have to keep the number of
> > dependencies down.
> >
> > -Jesper
> >
> >
> > On Wed, Feb 24, 2016 at 1:14 AM, Jesper Steen Møller <
> jesper@selskabet.org>
> > wrote:
> >>
> >> Hi Groovy-dev
> >>
> >> I’ve found some time to work on the Antlr4 parser and fixed some minor
> >> tasks there:
> >>
> >> * Fix grammar for several statement sequences, where NL or semicolon was
> >> not required (added negative test cases)
> >> * Remove non-reentrant static state in ASTBuilder
> >> * Added syntax error handling to Antlr 4 parser
> >> * Make sure GrammarPredicates work with arbitrarily nested package
> >> hierarchies
> >>
> >> I’m pushing my changes here:
> >> https://github.com/jespersm/groovy/commits/antlr4
> >>
> >> Next up is support for non-ID method invocation and command expressions,
> >> and then unary expressions.
> >> Also, since the GSoC is a bit dated, traits are also missing.
> >>
> >> -Jesper
> >
> >
> >
> >
> > --
> > Guillaume Laforge
> > Apache Groovy committer & PMC Vice-President
> > Product Ninja & Advocate at Restlet
> >
> > Blog: http://glaforge.appspot.com/
> > Social: @glaforge / Google+
> >
> >
>
Re: Antlr4-based Groovy parser status update
Posted by Paul King <pa...@asert.com.au>.
Spock for testing is by no means a show stopper!
On Wed, Feb 24, 2016 at 6:46 PM, Jesper Steen Møller
<je...@selskabet.org> wrote:
>
> On 24. feb. 2016, at 01.21, Guillaume Laforge <gl...@gmail.com> wrote:
>
> Sounds cool, Jesper, great to hear about your progress on the Antlr4
> grammar!
> Is there a way we can help you? How can we test drive this? Is the build
> working?
>
>
> The build works in that the antlr4 branch can be build and tested with e.g.
> “gradle -PuseAntlr4=true test”.
>
> However, I can’t seem to actually build a real distribution and use the new
> parser in that (there’s a dynamic switch), without getting some very strange
> errors.
> I’ve put off chasing that problem until I know I have the missing pieces of
> the grammar and the AST building done right.
>
> The current tests use Spock. Is that out of the question for the main Groovy
> build? I loooove Spock, but I realise we have to keep the number of
> dependencies down.
>
> -Jesper
>
>
> On Wed, Feb 24, 2016 at 1:14 AM, Jesper Steen Møller <je...@selskabet.org>
> wrote:
>>
>> Hi Groovy-dev
>>
>> I’ve found some time to work on the Antlr4 parser and fixed some minor
>> tasks there:
>>
>> * Fix grammar for several statement sequences, where NL or semicolon was
>> not required (added negative test cases)
>> * Remove non-reentrant static state in ASTBuilder
>> * Added syntax error handling to Antlr 4 parser
>> * Make sure GrammarPredicates work with arbitrarily nested package
>> hierarchies
>>
>> I’m pushing my changes here:
>> https://github.com/jespersm/groovy/commits/antlr4
>>
>> Next up is support for non-ID method invocation and command expressions,
>> and then unary expressions.
>> Also, since the GSoC is a bit dated, traits are also missing.
>>
>> -Jesper
>
>
>
>
> --
> Guillaume Laforge
> Apache Groovy committer & PMC Vice-President
> Product Ninja & Advocate at Restlet
>
> Blog: http://glaforge.appspot.com/
> Social: @glaforge / Google+
>
>
Re: More Antlr4-based Groovy parser status update
Posted by Guillaume Laforge <gl...@gmail.com>.
Yup, that would be great as metadata!
On Fri, Mar 11, 2016 at 11:26 AM, Jesper Steen Møller <je...@selskabet.org>
wrote:
> Hi Guillaume
>
> This is not covered by the current code, but it can be added. I guess this
> could be set as “metadata” on types, package, properties, methods, and
> fields?
>
> -Jesper
>
> On 11. mar. 2016, at 11.11, Guillaume Laforge <gl...@gmail.com> wrote:
>
> By the way, I had a question (unrelated to the below thread, but related
> to the grammar) :-)
>
> Do you keep the comment information?
> It's something we've always said we should support, and it would
> tremendously help making a less hackish groovydoc tool.
> Having AST nodes for JavaDoc comments would really be great.
>
> Guillaume
>
> On Sun, Feb 28, 2016 at 12:55 PM, Jesper Steen Møller <
> jesper@selskabet.org> wrote:
>
>> Hi Groovy-Dev
>>
>> Here’s another update on the progress on the Antlr4 parser, as maintained
>> on *https://github.com/jespersm/groovy.git
>> <https://github.com/jespersm/groovy.git>* (in the *antlr4* branch).
>> To play with it, try:
>>
>> $ git clone -b antlr4 https://jespersm@github.com/jespersm/groovy.git
>> $ cd groovy
>> $ gradle -PuseAntlr4=true console
>>
>>
>> I’ve fixed a number of issues:
>>
>> - Support method pointer operator
>> - Attributes/method/property names as strings/gstrings
>> - Real support for unary plus and minus (mimics old parser’s
>> behaviour)
>> - Compilation units not ending with semicolon or newline
>> - Slashy strings could span lines, confusing division statements and
>> comments
>>
>> I can now explore the new grammar and AST building using the Console,
>> which is fun, but it’s very easy to find unsupported constructs. Mapping
>> out the full Groovy grammar from the documentation alone is quite a task.
>> Just today, I discovered lacking support for ‘assert’ and for
>> ’super’-calls. The smaller issues currently are:
>>
>> - assert
>> - super()
>> - Full Unicode letter support for identifiers
>> - Support identifiers as property names and map literal entry names
>>
>>
>> The bigger issue is with converting the ASTBuilder to pure Java, a task I
>> havn’t started yet. Actually, this poses a different question for AST
>> generation: Whether to switch from tree-walking the parse tree (so whole
>> tree must be kept in memory), to the listener-based approach, where the AST
>> is built mostly bottom-up, ensuring smaller memory footprint.
>>
>> So you can help me with a couple of answers:
>>
>> - Memory: Is this an issue I should be focusing on — and is there a
>> test to baseline against?
>> - I’ve discovered a small issue with unary syntax. Currently, nested
>> unary expressions are not supported without parenthesis: Try e.g. - -1 or +
>> -1. Is this intentional, or just an artifact of the precedence-refactored
>> Java grammar?
>>
>>
>> -Jesper
>>
>>
>
>
> --
> Guillaume Laforge
> Apache Groovy committer & PMC Vice-President
> Product Ninja & Advocate at Restlet <http://restlet.com/>
>
> Blog: http://glaforge.appspot.com/
> Social: @glaforge <http://twitter.com/glaforge> / Google+
> <https://plus.google.com/u/0/114130972232398734985/posts>
>
>
>
--
Guillaume Laforge
Apache Groovy committer & PMC Vice-President
Product Ninja & Advocate at Restlet <http://restlet.com>
Blog: http://glaforge.appspot.com/
Social: @glaforge <http://twitter.com/glaforge> / Google+
<https://plus.google.com/u/0/114130972232398734985/posts>
Re: More Antlr4-based Groovy parser status update
Posted by Jesper Steen Møller <je...@selskabet.org>.
Hi Guillaume
This is not covered by the current code, but it can be added. I guess this could be set as “metadata” on types, package, properties, methods, and fields?
-Jesper
> On 11. mar. 2016, at 11.11, Guillaume Laforge <gl...@gmail.com> wrote:
>
> By the way, I had a question (unrelated to the below thread, but related to the grammar) :-)
>
> Do you keep the comment information?
> It's something we've always said we should support, and it would tremendously help making a less hackish groovydoc tool.
> Having AST nodes for JavaDoc comments would really be great.
>
> Guillaume
>
> On Sun, Feb 28, 2016 at 12:55 PM, Jesper Steen Møller <jesper@selskabet.org <ma...@selskabet.org>> wrote:
> Hi Groovy-Dev
>
> Here’s another update on the progress on the Antlr4 parser, as maintained on https://github.com/jespersm/groovy.git <https://github.com/jespersm/groovy.git> (in the antlr4 branch).
> To play with it, try:
>
> $ git clone -b antlr4 https://jespersm@github.com/jespersm/groovy.git <https://jespersm@github.com/jespersm/groovy.git>
> $ cd groovy
> $ gradle -PuseAntlr4=true console
>
> I’ve fixed a number of issues:
> Support method pointer operator
> Attributes/method/property names as strings/gstrings
> Real support for unary plus and minus (mimics old parser’s behaviour)
> Compilation units not ending with semicolon or newline
> Slashy strings could span lines, confusing division statements and comments
> I can now explore the new grammar and AST building using the Console, which is fun, but it’s very easy to find unsupported constructs. Mapping out the full Groovy grammar from the documentation alone is quite a task. Just today, I discovered lacking support for ‘assert’ and for ’super’-calls. The smaller issues currently are:
> assert
> super()
> Full Unicode letter support for identifiers
> Support identifiers as property names and map literal entry names
>
> The bigger issue is with converting the ASTBuilder to pure Java, a task I havn’t started yet. Actually, this poses a different question for AST generation: Whether to switch from tree-walking the parse tree (so whole tree must be kept in memory), to the listener-based approach, where the AST is built mostly bottom-up, ensuring smaller memory footprint.
>
> So you can help me with a couple of answers:
> Memory: Is this an issue I should be focusing on — and is there a test to baseline against?
> I’ve discovered a small issue with unary syntax. Currently, nested unary expressions are not supported without parenthesis: Try e.g. - -1 or + -1. Is this intentional, or just an artifact of the precedence-refactored Java grammar?
>
> -Jesper
>
>
>
>
> --
> Guillaume Laforge
> Apache Groovy committer & PMC Vice-President
> Product Ninja & Advocate at Restlet <http://restlet.com/>
>
> Blog: http://glaforge.appspot.com/ <http://glaforge.appspot.com/>
> Social: @glaforge <http://twitter.com/glaforge> / Google+ <https://plus.google.com/u/0/114130972232398734985/posts>
Re: More Antlr4-based Groovy parser status update
Posted by Jim Northrop <ja...@googlemail.com>.
+1 👏
Sent from my iPad
> On 11 Mar 2016, at 11:11, Guillaume Laforge <gl...@gmail.com> wrote:
>
> By the way, I had a question (unrelated to the below thread, but related to the grammar) :-)
>
> Do you keep the comment information?
> It's something we've always said we should support, and it would tremendously help making a less hackish groovydoc tool.
> Having AST nodes for JavaDoc comments would really be great.
>
> Guillaume
>
>> On Sun, Feb 28, 2016 at 12:55 PM, Jesper Steen Møller <je...@selskabet.org> wrote:
>> Hi Groovy-Dev
>>
>> Here’s another update on the progress on the Antlr4 parser, as maintained on https://github.com/jespersm/groovy.git (in the antlr4 branch).
>> To play with it, try:
>>
>> $ git clone -b antlr4 https://jespersm@github.com/jespersm/groovy.git
>> $ cd groovy
>> $ gradle -PuseAntlr4=true console
>>
>> I’ve fixed a number of issues:
>> Support method pointer operator
>> Attributes/method/property names as strings/gstrings
>> Real support for unary plus and minus (mimics old parser’s behaviour)
>> Compilation units not ending with semicolon or newline
>> Slashy strings could span lines, confusing division statements and comments
>> I can now explore the new grammar and AST building using the Console, which is fun, but it’s very easy to find unsupported constructs. Mapping out the full Groovy grammar from the documentation alone is quite a task. Just today, I discovered lacking support for ‘assert’ and for ’super’-calls. The smaller issues currently are:
>> assert
>> super()
>> Full Unicode letter support for identifiers
>> Support identifiers as property names and map literal entry names
>>
>> The bigger issue is with converting the ASTBuilder to pure Java, a task I havn’t started yet. Actually, this poses a different question for AST generation: Whether to switch from tree-walking the parse tree (so whole tree must be kept in memory), to the listener-based approach, where the AST is built mostly bottom-up, ensuring smaller memory footprint.
>>
>> So you can help me with a couple of answers:
>> Memory: Is this an issue I should be focusing on — and is there a test to baseline against?
>> I’ve discovered a small issue with unary syntax. Currently, nested unary expressions are not supported without parenthesis: Try e.g. - -1 or + -1. Is this intentional, or just an artifact of the precedence-refactored Java grammar?
>>
>> -Jesper
>
>
>
> --
> Guillaume Laforge
> Apache Groovy committer & PMC Vice-President
> Product Ninja & Advocate at Restlet
>
> Blog: http://glaforge.appspot.com/
> Social: @glaforge / Google+
Re: More Antlr4-based Groovy parser status update
Posted by Guillaume Laforge <gl...@gmail.com>.
By the way, I had a question (unrelated to the below thread, but related to
the grammar) :-)
Do you keep the comment information?
It's something we've always said we should support, and it would
tremendously help making a less hackish groovydoc tool.
Having AST nodes for JavaDoc comments would really be great.
Guillaume
On Sun, Feb 28, 2016 at 12:55 PM, Jesper Steen Møller <je...@selskabet.org>
wrote:
> Hi Groovy-Dev
>
> Here’s another update on the progress on the Antlr4 parser, as maintained
> on *https://github.com/jespersm/groovy.git
> <https://github.com/jespersm/groovy.git>* (in the *antlr4* branch).
> To play with it, try:
>
> $ git clone -b antlr4 https://jespersm@github.com/jespersm/groovy.git
> $ cd groovy
> $ gradle -PuseAntlr4=true console
>
>
> I’ve fixed a number of issues:
>
> - Support method pointer operator
> - Attributes/method/property names as strings/gstrings
> - Real support for unary plus and minus (mimics old parser’s behaviour)
> - Compilation units not ending with semicolon or newline
> - Slashy strings could span lines, confusing division statements and
> comments
>
> I can now explore the new grammar and AST building using the Console,
> which is fun, but it’s very easy to find unsupported constructs. Mapping
> out the full Groovy grammar from the documentation alone is quite a task.
> Just today, I discovered lacking support for ‘assert’ and for
> ’super’-calls. The smaller issues currently are:
>
> - assert
> - super()
> - Full Unicode letter support for identifiers
> - Support identifiers as property names and map literal entry names
>
>
> The bigger issue is with converting the ASTBuilder to pure Java, a task I
> havn’t started yet. Actually, this poses a different question for AST
> generation: Whether to switch from tree-walking the parse tree (so whole
> tree must be kept in memory), to the listener-based approach, where the AST
> is built mostly bottom-up, ensuring smaller memory footprint.
>
> So you can help me with a couple of answers:
>
> - Memory: Is this an issue I should be focusing on — and is there a
> test to baseline against?
> - I’ve discovered a small issue with unary syntax. Currently, nested
> unary expressions are not supported without parenthesis: Try e.g. - -1 or +
> -1. Is this intentional, or just an artifact of the precedence-refactored
> Java grammar?
>
>
> -Jesper
>
>
--
Guillaume Laforge
Apache Groovy committer & PMC Vice-President
Product Ninja & Advocate at Restlet <http://restlet.com>
Blog: http://glaforge.appspot.com/
Social: @glaforge <http://twitter.com/glaforge> / Google+
<https://plus.google.com/u/0/114130972232398734985/posts>
Re: More Antlr4-based Groovy parser status update
Posted by Jesper Steen Møller <je...@selskabet.org>.
Just a clarification:
I wrote:
> I’ve discovered a small issue with unary syntax. Currently, nested unary expressions are not supported without parenthesis: Try e.g. - -1 or + -1. Is this intentional, or just an artifact of the precedence-refactored Java grammar?
>
By “currently”, I mean in Groovy 2.x. In Groovy 2.x “- -1” is illegal. In Java, it’s allowed - and equals 1. I propose that the new grammar should allow this, too.
Any objections?
-Jesper
Re: More Antlr4-based Groovy parser status update
Posted by Jesper Steen Møller <je...@selskabet.org>.
Hi Jochen
> On 29. feb. 2016, at 11.39, Jochen Theodorou <bl...@gmx.org> wrote:
>
> On 28.02.2016 12:55, Jesper Steen Møller wrote:
> [...]
>> * Memory: Is this an issue I should be focusing on — and is there a
>> test to baseline against?
>
> no test you can baseline against imho. The current approach is keeping everything in memory and uses that for the cst part, which is build in the parsing phase of the compiler. The conversion to an AST is done in the next step, the conversion step.
>
I’ll find a way to benchmark this with the existing test suite using ThreadMXBean.getThreadAllocatedBytes, such as total bytes allocated for some big Groovy files, once the grammar is complete. Antlr4 is pretty lean on resource use, if used correctly.
When compiling several files, are multiple AST’s in memory, or how does that work? If so, we’d also have to ensure that the live set of the AST doesn’t grow.
> I my opinion the cst and the getter for the cst should be deprecated... but this may require some changes in the compiler and its infrastructure… so in total, the tree approach might be easier for now.
>
You read my mind, I was wondering if it’d be OK to get rid of the CST part. I don’t see why we can’t parse straight to AST.
Unfortunately, it’s not possible to eliminate the CSTNode hierarchy without changing the AST’s API (for Token, used for differentiating between binary ops) and LocatedMessage API (for CSTNode), and that would break compatibility.
I’m not sure if Groovy 3.0 would allow such API breakage, so I’m assuming ’no’.
>> * I’ve discovered a small issue with unary syntax. Currently, nested
>> unary expressions are not supported without parenthesis: Try e.g. -
>> -1 or + -1. Is this intentional, or just an artifact of the
>> precedence-refactored Java grammar?
>
>
> you mean "- -1" as in -(-1)? I think there was a grammar reason for this. If it there is no trouble with this in the new grammar, then you can support it, np.. I see that as an optional
Yes, exactly. New grammar supports this.
Thanks for your answers!
-Jesper
Re: More Antlr4-based Groovy parser status update
Posted by Jochen Theodorou <bl...@gmx.org>.
On 28.02.2016 12:55, Jesper Steen Møller wrote:
[...]
> * Memory: Is this an issue I should be focusing on — and is there a
> test to baseline against?
no test you can baseline against imho. The current approach is keeping
everything in memory and uses that for the cst part, which is build in
the parsing phase of the compiler. The conversion to an AST is done in
the next step, the conversion step.
I my opinion the cst and the getter for the cst should be deprecated...
but this may require some changes in the compiler and its
infrastructure... so in total, the tree approach might be easier for now.
> * I’ve discovered a small issue with unary syntax. Currently, nested
> unary expressions are not supported without parenthesis: Try e.g. -
> -1 or + -1. Is this intentional, or just an artifact of the
> precedence-refactored Java grammar?
you mean "- -1" as in -(-1)? I think there was a grammar reason for
this. If it there is no trouble with this in the new grammar, then you
can support it, np.. I see that as an optional
bye Jochen
Re: More Antlr4-based Groovy parser status update
Posted by Jesper Steen Møller <je...@selskabet.org>.
Hi Pascal,
Thank you, I’d missed that one.
Any thought on the validity of “- -1”?
-Jesper
> On 28. feb. 2016, at 18.23, Pascal Schumacher <pascalschumacher@gmx.net <ma...@gmx.net>> wrote:
>
> Hi Jesper,
>
> thanks for the update. :) Nice to hear you are progressing.
>
> Concerning the ASTBuilder to Java conversion, there is a pull request with this at the old repo https://github.com/groovy/groovy-core/pull/513 <https://github.com/groovy/groovy-core/pull/513>
>
> Cheers,
> Pascal
>
> Am 28.02.2016 um 12:55 schrieb Jesper Steen Møller:
>> Hi Groovy-Dev
>>
>> Here’s another update on the progress on the Antlr4 parser, as maintained on https://github.com/jespersm/groovy.git <https://github.com/jespersm/groovy.git> (in the antlr4 branch).
>> To play with it, try:
>>
>> $ git clone -b antlr4 https://jespersm@github.com/jespersm/groovy.git <https://jespersm@github.com/jespersm/groovy.git>
>> $ cd groovy
>> $ gradle -PuseAntlr4=true console
>>
>> I’ve fixed a number of issues:
>> Support method pointer operator
>> Attributes/method/property names as strings/gstrings
>> Real support for unary plus and minus (mimics old parser’s behaviour)
>> Compilation units not ending with semicolon or newline
>> Slashy strings could span lines, confusing division statements and comments
>> I can now explore the new grammar and AST building using the Console, which is fun, but it’s very easy to find unsupported constructs. Mapping out the full Groovy grammar from the documentation alone is quite a task. Just today, I discovered lacking support for ‘assert’ and for ’super’-calls. The smaller issues currently are:
>> assert
>> super()
>> Full Unicode letter support for identifiers
>> Support identifiers as property names and map literal entry names
>>
>> The bigger issue is with converting the ASTBuilder to pure Java, a task I havn’t started yet. Actually, this poses a different question for AST generation: Whether to switch from tree-walking the parse tree (so whole tree must be kept in memory), to the listener-based approach, where the AST is built mostly bottom-up, ensuring smaller memory footprint.
>>
>> So you can help me with a couple of answers:
>> Memory: Is this an issue I should be focusing on — and is there a test to baseline against?
>> I’ve discovered a small issue with unary syntax. Currently, nested unary expressions are not supported without parenthesis: Try e.g. - -1 or + -1. Is this intentional, or just an artifact of the precedence-refactored Java grammar?
>>
>> -Jesper
>>
>
Re: More Antlr4-based Groovy parser status update
Posted by Pascal Schumacher <pa...@gmx.net>.
Hi Jesper,
thanks for the update. :) Nice to hear you are progressing.
Concerning the ASTBuilder to Java conversion, there is a pull request
with this at the old repo https://github.com/groovy/groovy-core/pull/513
Cheers,
Pascal
Am 28.02.2016 um 12:55 schrieb Jesper Steen Møller:
> Hi Groovy-Dev
>
> Here’s another update on the progress on the Antlr4 parser, as
> maintained on *https://github.com/jespersm/groovy.git* (in the
> *antlr4* branch).
> To play with it, try:
>
> $ git clone -b antlr4 https://jespersm@github.com/jespersm/groovy.git
> $ cd groovy
> $ gradle -PuseAntlr4=true console
>
>
> I’ve fixed a number of issues:
>
> * Support method pointer operator
> * Attributes/method/property names as strings/gstrings
> * Real support for unary plus and minus (mimics old parser’s behaviour)
> * Compilation units not ending with semicolon or newline
> * Slashy strings could span lines, confusing division statements and
> comments
>
> I can now explore the new grammar and AST building using the Console,
> which is fun, but it’s very easy to find unsupported constructs.
> Mapping out the full Groovy grammar from the documentation alone is
> quite a task. Just today, I discovered lacking support for ‘assert’
> and for ’super’-calls. The smaller issues currently are:
>
> * assert
> * super()
> * Full Unicode letter support for identifiers
> * Support identifiers as property names and map literal entry names
>
>
> The bigger issue is with converting the ASTBuilder to pure Java, a
> task I havn’t started yet. Actually, this poses a different question
> for AST generation: Whether to switch from tree-walking the parse tree
> (so whole tree must be kept in memory), to the listener-based
> approach, where the AST is built mostly bottom-up, ensuring smaller
> memory footprint.
>
> So you can help me with a couple of answers:
>
> * Memory: Is this an issue I should be focusing on — and is there a
> test to baseline against?
> * I’ve discovered a small issue with unary syntax. Currently, nested
> unary expressions are not supported without parenthesis: Try e.g.
> - -1 or + -1. Is this intentional, or just an artifact of the
> precedence-refactored Java grammar?
>
>
> -Jesper
>
More Antlr4-based Groovy parser status update
Posted by Jesper Steen Møller <je...@selskabet.org>.
Hi Groovy-Dev
Here’s another update on the progress on the Antlr4 parser, as maintained on https://github.com/jespersm/groovy.git (in the antlr4 branch).
To play with it, try:
$ git clone -b antlr4 https://jespersm@github.com/jespersm/groovy.git
$ cd groovy
$ gradle -PuseAntlr4=true console
I’ve fixed a number of issues:
Support method pointer operator
Attributes/method/property names as strings/gstrings
Real support for unary plus and minus (mimics old parser’s behaviour)
Compilation units not ending with semicolon or newline
Slashy strings could span lines, confusing division statements and comments
I can now explore the new grammar and AST building using the Console, which is fun, but it’s very easy to find unsupported constructs. Mapping out the full Groovy grammar from the documentation alone is quite a task. Just today, I discovered lacking support for ‘assert’ and for ’super’-calls. The smaller issues currently are:
assert
super()
Full Unicode letter support for identifiers
Support identifiers as property names and map literal entry names
The bigger issue is with converting the ASTBuilder to pure Java, a task I havn’t started yet. Actually, this poses a different question for AST generation: Whether to switch from tree-walking the parse tree (so whole tree must be kept in memory), to the listener-based approach, where the AST is built mostly bottom-up, ensuring smaller memory footprint.
So you can help me with a couple of answers:
Memory: Is this an issue I should be focusing on — and is there a test to baseline against?
I’ve discovered a small issue with unary syntax. Currently, nested unary expressions are not supported without parenthesis: Try e.g. - -1 or + -1. Is this intentional, or just an artifact of the precedence-refactored Java grammar?
-Jesper
Re: Antlr4-based Groovy parser status update
Posted by Jesper Steen Møller <je...@selskabet.org>.
> On 24. feb. 2016, at 01.21, Guillaume Laforge <gl...@gmail.com> wrote:
>
> Sounds cool, Jesper, great to hear about your progress on the Antlr4 grammar!
> Is there a way we can help you? How can we test drive this? Is the build working?
The build works in that the antlr4 branch can be build and tested with e.g. “gradle -PuseAntlr4=true test”.
However, I can’t seem to actually build a real distribution and use the new parser in that (there’s a dynamic switch), without getting some very strange errors.
I’ve put off chasing that problem until I know I have the missing pieces of the grammar and the AST building done right.
The current tests use Spock. Is that out of the question for the main Groovy build? I loooove Spock, but I realise we have to keep the number of dependencies down.
-Jesper
>
> On Wed, Feb 24, 2016 at 1:14 AM, Jesper Steen Møller <jesper@selskabet.org <ma...@selskabet.org>> wrote:
> Hi Groovy-dev
>
> I’ve found some time to work on the Antlr4 parser and fixed some minor tasks there:
>
> * Fix grammar for several statement sequences, where NL or semicolon was not required (added negative test cases)
> * Remove non-reentrant static state in ASTBuilder
> * Added syntax error handling to Antlr 4 parser
> * Make sure GrammarPredicates work with arbitrarily nested package hierarchies
>
> I’m pushing my changes here:
> https://github.com/jespersm/groovy/commits/antlr4 <https://github.com/jespersm/groovy/commits/antlr4>
>
> Next up is support for non-ID method invocation and command expressions, and then unary expressions.
> Also, since the GSoC is a bit dated, traits are also missing.
>
> -Jesper
>
>
>
> --
> Guillaume Laforge
> Apache Groovy committer & PMC Vice-President
> Product Ninja & Advocate at Restlet <http://restlet.com/>
>
> Blog: http://glaforge.appspot.com/ <http://glaforge.appspot.com/>
> Social: @glaforge <http://twitter.com/glaforge> / Google+ <https://plus.google.com/u/0/114130972232398734985/posts>
Re: Antlr4-based Groovy parser status update
Posted by Guillaume Laforge <gl...@gmail.com>.
Sounds cool, Jesper, great to hear about your progress on the Antlr4
grammar!
Is there a way we can help you? How can we test drive this? Is the build
working?
Guillaume
On Wed, Feb 24, 2016 at 1:14 AM, Jesper Steen Møller <je...@selskabet.org>
wrote:
> Hi Groovy-dev
>
> I’ve found some time to work on the Antlr4 parser and fixed some minor
> tasks there:
>
> * Fix grammar for several statement sequences, where NL or semicolon was
> not required (added negative test cases)
> * Remove non-reentrant static state in ASTBuilder
> * Added syntax error handling to Antlr 4 parser
> * Make sure GrammarPredicates work with arbitrarily nested package
> hierarchies
>
> I’m pushing my changes here:
> https://github.com/jespersm/groovy/commits/antlr4
>
> Next up is support for non-ID method invocation and command expressions,
> and then unary expressions.
> Also, since the GSoC is a bit dated, traits are also missing.
>
> -Jesper
--
Guillaume Laforge
Apache Groovy committer & PMC Vice-President
Product Ninja & Advocate at Restlet <http://restlet.com>
Blog: http://glaforge.appspot.com/
Social: @glaforge <http://twitter.com/glaforge> / Google+
<https://plus.google.com/u/0/114130972232398734985/posts>
Re: Antlr4-based Groovy parser status update
Posted by daniel_sun <re...@hotmail.com>.
Hi Jesper,
I'm very glad to know the antlr4 works on Groovy re-launched.
A few days ago, I gave it a try and found it did not support optional
parentheses of invoking method, so I managed to fix it and new a pull
request on the github, pls verify it.
The dev environment setting up makes me painful. I have no idea how
to run test cases on antlr4 only(running all test cases is too time
consuming to develop efficiently). I use IntelliJ IDEA15 to contribute to
Groovy. Could you tell me how to set up dev environment and run test cases
on antlr4 only?
Thanks in advance
Daniel.Sun
--
View this message in context: http://groovy.329449.n5.nabble.com/Antlr4-based-Groovy-parser-status-update-tp5731370p5731890.html
Sent from the Groovy Dev mailing list archive at Nabble.com.