Re: recommendation for language support

[Tim Boudreau <ni...@gmail.com>]

I'd strongly recommend going with Antlr-4 - it is very nice.

There is one impedance mismatch to be aware of, to save a few hours of
cursing both Antlr and NetBeans' lexer infrastructure: Antlr may give you
an EOF token that is non-empty - check it, or you won't consume all the
characters from the input, and if you don't, an exception will be thrown
saying you didn't consume all the characters. The LexerInput you get passed
has to get entirely consumed before you return a null to indicate you've
lexed the entire input. If it wasn't, the infrastructure assumes that's a
bug.

One other non-obvious thing: If your grammar users channels to ignore
whitespace or comments when parsing, make sure:
 - that when you're lexing you use a stream that returns ALL the tokens on
ALL channels (if using CommonTokenStream pass -1 for the channel, if you're
wrappering LexerInput, you're all set)
 - conversely, when parsing, make sure you use a stream that only returns
channel 0 tokens, or you will think your parser is broken when it isn't)

Having written several modules that add language support using Antlr, I've
considered factoring some of the always the same stuff (generating token
id's from a vocabulary, stuff like that) into a support module.

Over the past few weeks I've been doing some heavy duty work on the
NetBeans Antlr plugin in github, which appears to be abandoned and was in
need of a few features - like being able to syntax highlight files in a
language under development and do error highlighting of them, and refresh
that when the grammar is edited (which involves a level of hackery
bordering in evil, since it means working around all of the declarative
mechanisms for languages, data loaders, mime resolvers and everything else
to be installed, to allow something to imperatively make languages appear
out of the ether with all their editor plumbing, then change, then vanish).
When there's something reasonably non fragile to test, I'll post here.

I'd suggest using Maven for your plugin, as the Maven Antlr plugin is
pretty easy to set up, and results in a pretty easy to understand structure
for things.

-Tim

On Tue, Oct 30, 2018 at 3:37 PM Mario Schroeder <ma...@gmail.com>
wrote:

> Hi,
>
> I would appreciate when someone could set me on the right track. I'm
> wondering what is the best way to write a plugin with a support for a new
> language.
>
> I have found this old tutorial:
> http://wiki.netbeans.org/How_to_create_support_for_a_new_language
> It uses JavaCC. The tutorial is linked to another one, which makes use of
> ANTLR. When I have a look at NetBeans source code, there are some templates
> written with JFlex. So I'm confused. Which one should I use?
>
> I would prefer ANTLR, since there seems to be more documentation for it.
> Does anyone have a recommendation?
>
> Regards,
> Mario
>
-- 
http://timboudreau.com

Re: recommendation for language support

[Jean-Marc Borer <jm...@gmail.com>]

Very interesting topic.

As it seems there are people around with knowledge of AST and source
parsing, I would like to ask if someone knows how I can get the java source
AST (property in a Document?) and how I can listen when it is updated? I am
currently working on a plugin that will handle color preview in the Java
source code.

Cheers.

JMB


On Wed, Nov 7, 2018 at 4:32 PM Mario Schroeder <ma...@gmail.com>
wrote:

> Hello NetBeaners,
>
> I finished my grammar based on ANTLR. The current version satisfies me.
> However since it is my first attempt I would highly appreciate if someone
> from this thread could have a look and comment if something could be
> improved. I know that you guys might be busy with the new release, so I
> will be very thankful for a little bit of your time.
>
> The the grammar can be found here:
> https://github.com/mario-s/grammars-v4/tree/cto/cto
>
> Thanks in advance!
> -Mario
>
> Am So., 4. Nov. 2018 um 05:13 Uhr schrieb Tim Boudreau <
> niftiness@gmail.com
> >:
>
> > I'd strongly recommend going with Antlr-4 - it is very nice.
> >
> > There is one impedance mismatch to be aware of, to save a few hours of
> > cursing both Antlr and NetBeans' lexer infrastructure: Antlr may give you
> > an EOF token that is non-empty - check it, or you won't consume all the
> > characters from the input, and if you don't, an exception will be thrown
> > saying you didn't consume all the characters. The LexerInput you get
> passed
> > has to get entirely consumed before you return a null to indicate you've
> > lexed the entire input. If it wasn't, the infrastructure assumes that's a
> > bug.
> >
> > One other non-obvious thing: If your grammar users channels to ignore
> > whitespace or comments when parsing, make sure:
> >  - that when you're lexing you use a stream that returns ALL the tokens
> on
> > ALL channels (if using CommonTokenStream pass -1 for the channel, if
> you're
> > wrappering LexerInput, you're all set)
> >  - conversely, when parsing, make sure you use a stream that only returns
> > channel 0 tokens, or you will think your parser is broken when it isn't)
> >
> > Having written several modules that add language support using Antlr,
> I've
> > considered factoring some of the always the same stuff (generating token
> > id's from a vocabulary, stuff like that) into a support module.
> >
> > Over the past few weeks I've been doing some heavy duty work on the
> > NetBeans Antlr plugin in github, which appears to be abandoned and was in
> > need of a few features - like being able to syntax highlight files in a
> > language under development and do error highlighting of them, and refresh
> > that when the grammar is edited (which involves a level of hackery
> > bordering in evil, since it means working around all of the declarative
> > mechanisms for languages, data loaders, mime resolvers and everything
> else
> > to be installed, to allow something to imperatively make languages appear
> > out of the ether with all their editor plumbing, then change, then
> vanish).
> > When there's something reasonably non fragile to test, I'll post here.
> >
> > I'd suggest using Maven for your plugin, as the Maven Antlr plugin is
> > pretty easy to set up, and results in a pretty easy to understand
> structure
> > for things.
> >
> > -Tim
> >
> > On Tue, Oct 30, 2018 at 3:37 PM Mario Schroeder <ma...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I would appreciate when someone could set me on the right track. I'm
> > > wondering what is the best way to write a plugin with a support for a
> new
> > > language.
> > >
> > > I have found this old tutorial:
> > > http://wiki.netbeans.org/How_to_create_support_for_a_new_language
> > > It uses JavaCC. The tutorial is linked to another one, which makes use
> of
> > > ANTLR. When I have a look at NetBeans source code, there are some
> > templates
> > > written with JFlex. So I'm confused. Which one should I use?
> > >
> > > I would prefer ANTLR, since there seems to be more documentation for
> it.
> > > Does anyone have a recommendation?
> > >
> > > Regards,
> > > Mario
> > >
> > --
> > http://timboudreau.com
> >
>

Re: recommendation for language support

[Mario Schroeder <ma...@gmail.com>]

Hello NetBeaners,

I finished my grammar based on ANTLR. The current version satisfies me.
However since it is my first attempt I would highly appreciate if someone
from this thread could have a look and comment if something could be
improved. I know that you guys might be busy with the new release, so I
will be very thankful for a little bit of your time.

The the grammar can be found here:
https://github.com/mario-s/grammars-v4/tree/cto/cto

Thanks in advance!
-Mario

Am So., 4. Nov. 2018 um 05:13 Uhr schrieb Tim Boudreau <niftiness@gmail.com
>:

> I'd strongly recommend going with Antlr-4 - it is very nice.
>
> There is one impedance mismatch to be aware of, to save a few hours of
> cursing both Antlr and NetBeans' lexer infrastructure: Antlr may give you
> an EOF token that is non-empty - check it, or you won't consume all the
> characters from the input, and if you don't, an exception will be thrown
> saying you didn't consume all the characters. The LexerInput you get passed
> has to get entirely consumed before you return a null to indicate you've
> lexed the entire input. If it wasn't, the infrastructure assumes that's a
> bug.
>
> One other non-obvious thing: If your grammar users channels to ignore
> whitespace or comments when parsing, make sure:
>  - that when you're lexing you use a stream that returns ALL the tokens on
> ALL channels (if using CommonTokenStream pass -1 for the channel, if you're
> wrappering LexerInput, you're all set)
>  - conversely, when parsing, make sure you use a stream that only returns
> channel 0 tokens, or you will think your parser is broken when it isn't)
>
> Having written several modules that add language support using Antlr, I've
> considered factoring some of the always the same stuff (generating token
> id's from a vocabulary, stuff like that) into a support module.
>
> Over the past few weeks I've been doing some heavy duty work on the
> NetBeans Antlr plugin in github, which appears to be abandoned and was in
> need of a few features - like being able to syntax highlight files in a
> language under development and do error highlighting of them, and refresh
> that when the grammar is edited (which involves a level of hackery
> bordering in evil, since it means working around all of the declarative
> mechanisms for languages, data loaders, mime resolvers and everything else
> to be installed, to allow something to imperatively make languages appear
> out of the ether with all their editor plumbing, then change, then vanish).
> When there's something reasonably non fragile to test, I'll post here.
>
> I'd suggest using Maven for your plugin, as the Maven Antlr plugin is
> pretty easy to set up, and results in a pretty easy to understand structure
> for things.
>
> -Tim
>
> On Tue, Oct 30, 2018 at 3:37 PM Mario Schroeder <ma...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I would appreciate when someone could set me on the right track. I'm
> > wondering what is the best way to write a plugin with a support for a new
> > language.
> >
> > I have found this old tutorial:
> > http://wiki.netbeans.org/How_to_create_support_for_a_new_language
> > It uses JavaCC. The tutorial is linked to another one, which makes use of
> > ANTLR. When I have a look at NetBeans source code, there are some
> templates
> > written with JFlex. So I'm confused. Which one should I use?
> >
> > I would prefer ANTLR, since there seems to be more documentation for it.
> > Does anyone have a recommendation?
> >
> > Regards,
> > Mario
> >
> --
> http://timboudreau.com
>