You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by antonio <an...@vieiro.net> on 2021/05/06 06:52:05 UTC
[DISCUSS] CND, cnd.antlr & cnd.apt on top of Antlrv4?
Hi all,
As you may remember from previous threads, the NetBeans 8.2 "cnd.antlr"
module contained a fork of Antlr 2.7.7 (compiler & runtime) that was not
donated to the ASF. As a substitute Oracle donated a
"cnd/libs.cnd.antlr" library wrapper that wrapped the NetBeans 8.2 jar,
that we cannot use in Apache NetBeans due to licensing problems.
The "cnd.apt" package contains an old "aptlexer.g" (on top of this Antlr
2.7.7 runtime) lexer grammar that is responsible for lexing C/C++ and
Fortran files.
I've done a small experiment (see [1]) transforming this old "aptexpr.g"
grammar with 1403 lines [2] to a brand new "APTLexer.g4" lexer grammar
[3] with 480 lines.
This is still on early stages, but it seems the resulting Antlrv4 lexer
outperforms the old Antlrv2 lexer (we go from ~2600 files/second to
~4000 files/second).
I think we can continue the path of removing the old "libs.antlr.cnd"
module from CND and then refactor "cnd.apt" (and "cnd.asm" et al) on top
of the new lexer and Antlrv4. The idea is to upgrade all existing
grammars (including the CXX parser at
https://github.com/apache/netbeans/blob/cnd/cnd/cnd.modelimpl/src/org/netbeans/modules/cnd/modelimpl/parser/CXXParser.g)
to Antlrv4 on top of this new lexer.
Since the "libs.antlr.cnd" API permeates different modules of CND this
may take quite a long time.
I think we can create a brand new "cnd2" branch in the main repository
(forked from the "cnd" branch and refreshed with latest 12.4 version)
and do this CND refactoring on top of Antlrv4 there, right? Any
objections to this? Or shall I continue this experiment in a repo of
mine? Any other plans/suggestions to have CND back?
Thanks,
Antonio
P.S.: In order to test the lexer I'd appreciate if you run the new lexer
(instructions at [1]) on some of your include/source directories and
report back any syntax errors you may see (the lexer will halt on any
lexer error), so we can fine-tune any extra lexer rules we may need.
[1]
https://github.com/vieiro/cnd.nextapt
To lex a whole directory of (*.h/*.c files) run the command:
java -Dlexer.print=false -jar
target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include/
To lex a specific file (dumping all tokens to stdout by removing
-Dlexer.print=false):
java -jar target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar
your-file.[ch]
[2]
https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.nb82/aptlexer.g
[3]
https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.antlr4/src/main/antlr4/org/netbeans/modules/cnd/apt/impl/support/generated/APTLexer.g4
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [DISCUSS] CND, cnd.antlr & cnd.apt on top of Antlrv4?
Posted by John Kostaras <jk...@gmail.com>.
sorry I didn't. I did know and seems ok. Will continue testing... Great
work.
Thanks.
John.
On Fri, 14 May 2021 at 18:02, antonio <an...@vieiro.net> wrote:
> Hi John,
>
> Did you git pull first? This changes quickly! :-).
>
> Thanks,
> Antonio
>
> El 14/5/21 a las 15:50, John Kostaras escribió:
> > Hallo Antonio,
> >
> > so, basically the error is that it cannot handle escape characters, e.g.
> \".
> >
> >
> > Here are some tests:
> >
> > Syntax error on 7613:20 token recognition error at: '" <?xml version=\"'
> >
> > Syntax error on 3743:20 token recognition error at: '"upd_cmd desc: \"'
> >
> > Syntax error on 523:3 token recognition error at: '"Error opening file.
> > PATH: \"'
> >
> > sprintf (trace_msg,
> >
> > "Error opening file. PATH: \"%s\" FILE: \"%s\"",
> >
> > getenv ((char *) PATH), afilename);
> >
> >
> >
> > Syntax error on 628:37 token recognition error at: '"Cannot get total
> > number of bytes from message \"'
> >
> > snprintf(errMsg, MY_ERR_LEN, "Cannot get total number of bytes from
> > message \"%s\"", sizBuf);
> >
> >
> >
> > Syntax error on 856:8 token recognition error at: '"<dto-ref ID = \"'
> >
> > Syntax error on 80:22 token recognition error at: '"Object 1:
> > Name: \"'
> >
> > str[0] = (char *) "Object 1: Name: \"OS\"\n";
> >
> > Syntax error on 29:9 token recognition error at: '"HTML <h3><li><a
> name=\"'
> >
> > Syntax error on 388:6 token recognition error at: '"%s DATA
> > user_id = \"'
> >
> > Syntax error on 180:12 token recognition error at: '"first line = \"'
> >
> > Syntax error on 155:8 token recognition error at: '"#include \"'
> >
> > fputs("#include \"common_types.h\"\n", errorMsg);
> >
> > Syntax error on 245:3 token recognition error at: '"Host list: \"'
> >
> > Syntax error on 499:11 token recognition error at: '"\CreateUser (\"'
> >
> > Syntax error on 67:9 token recognition error at: '"uptime |awk '{print
> \"'
> >
> > Syntax error on 51:14 token recognition error at: '"s=\"'
> >
> > Syntax error on 41:55 token recognition error at: '"00\"'
> >
> > Syntax error on 3040:4 token recognition error at: '" \"'
> >
> > Kind regards,
> >
> > John.
> >
> > On Wed, 12 May 2021 at 00:11, antonio <an...@vieiro.net> wrote:
> >
> >> Thanks John,
> >>
> >> I'll be adding assorted compiler-specific cases at [1].
> >>
> >> Kind regards,
> >> Antonio
> >>
> >>
> >> [1]
> >>
> >>
> https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.antlr4/src/test/resources/org/netbeans/modules/cnd/nextapt/antlr4/c/regressions.c
> >>
> >> On 11/05/2021 17:41, John Kostaras wrote:
> >>> #define CMPLX(__real,__imag) \
> >>> _Pragma("clang diagnostic push") \
> >>> * _Pragma("clang diagnostic ignored \"-Wcomplex-component-init\"")
> \*
> >>> (double _Complex){(__real),(__imag)} \
> >>> _Pragma("clang diagnostic pop")
> >>>
> >>> line 59 being the highlighted one.
> >>>
> >>> After pulling, building, re-running: java -Dlexer.print=false -jar
> >>> target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar
> /usr/include
> >>>
> >>> I get
> >>>
> >>> -- /usr/include/complex.h --
> >>> Syntax error on 59:12 token recognition error at: '"clang diagnostic
> >>> ignored \"'
> >>> Please report this error on the dev mailing list, including the line
> >> where
> >>> the error happeened.%n
> >>>
> >>>
> >>>
> >>> On Mon, 10 May 2021 at 23:58, antonio <an...@vieiro.net> wrote:
> >>>
> >>>> Hi John,
> >>>>
> >>>> Thanks for the report! It seems we weren't lexing '#error' yet (that's
> >>>> probably what is on line 59 in "/usr/include/complex.h", right?
> >>>> Otherwise send me the line 59).
> >>>>
> >>>> The newest version should detect #error properly (and also checks for
> >>>> ends of directives, and *.hpp/*.cpp).
> >>>>
> >>>> Talking about antlr4, it seems the AST is analyzed using the visitor
> >>>> pattern (that approach is also used in libclang) but NetBeans <= 8.2
> >>>> expects the whole AST in memory, so it seems we have a long
> refactoring
> >>>> road ahead (all this after successfully migrating the CXXParser.g to
> >>>> antlr4, which reads >3300 lines long).
> >>>>
> >>>> In parallel I'm giving libclang [1] a run. It seems quite powerful (it
> >>>> provides completion, precompiled headers, reparsing, diagnostics, AST)
> >>>> but it may require an extra +30Mb download on Windows (linux users may
> >>>> apt-get install libclang, though). More on this in a few days.
> >>>>
> >>>> Thanks again,
> >>>> Antonio
> >>>>
> >>>> [1]
> >>>> https://clang.llvm.org/doxygen/group__CINDEX.html
> >>>>
> >>>> El 10/5/21 a las 21:39, John Kostaras escribió:
> >>>>> +1 cnd.nextapt
> >>>>>
> >>>>> java -Dlexer.print=false -jar
> >>>>> target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar
> >> /usr/include
> >>>>> ....
> >>>>> -- /usr/include/complex.h --
> >>>>> Syntax error on 59:12 token recognition error at: '"clang diagnostic
> >>>>> ignored \"'
> >>>>>
> >>>>> Will also take a look. Thanks Antonio. Great initiative.
> >>>>>
> >>>>> P.S. I guess .hpp/.cpp files are next?
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> >>>> For additional commands, e-mail: dev-help@netbeans.apache.org
> >>>>
> >>>> For further information about the NetBeans mailing lists, visit:
> >>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> >> For additional commands, e-mail: dev-help@netbeans.apache.org
> >>
> >> For further information about the NetBeans mailing lists, visit:
> >> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
> >>
> >>
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
Re: [DISCUSS] CND, cnd.antlr & cnd.apt on top of Antlrv4?
Posted by antonio <an...@vieiro.net>.
Hi John,
Did you git pull first? This changes quickly! :-).
Thanks,
Antonio
El 14/5/21 a las 15:50, John Kostaras escribió:
> Hallo Antonio,
>
> so, basically the error is that it cannot handle escape characters, e.g. \".
>
>
> Here are some tests:
>
> Syntax error on 7613:20 token recognition error at: '" <?xml version=\"'
>
> Syntax error on 3743:20 token recognition error at: '"upd_cmd desc: \"'
>
> Syntax error on 523:3 token recognition error at: '"Error opening file.
> PATH: \"'
>
> sprintf (trace_msg,
>
> "Error opening file. PATH: \"%s\" FILE: \"%s\"",
>
> getenv ((char *) PATH), afilename);
>
>
>
> Syntax error on 628:37 token recognition error at: '"Cannot get total
> number of bytes from message \"'
>
> snprintf(errMsg, MY_ERR_LEN, "Cannot get total number of bytes from
> message \"%s\"", sizBuf);
>
>
>
> Syntax error on 856:8 token recognition error at: '"<dto-ref ID = \"'
>
> Syntax error on 80:22 token recognition error at: '"Object 1:
> Name: \"'
>
> str[0] = (char *) "Object 1: Name: \"OS\"\n";
>
> Syntax error on 29:9 token recognition error at: '"HTML <h3><li><a name=\"'
>
> Syntax error on 388:6 token recognition error at: '"%s DATA
> user_id = \"'
>
> Syntax error on 180:12 token recognition error at: '"first line = \"'
>
> Syntax error on 155:8 token recognition error at: '"#include \"'
>
> fputs("#include \"common_types.h\"\n", errorMsg);
>
> Syntax error on 245:3 token recognition error at: '"Host list: \"'
>
> Syntax error on 499:11 token recognition error at: '"\CreateUser (\"'
>
> Syntax error on 67:9 token recognition error at: '"uptime |awk '{print \"'
>
> Syntax error on 51:14 token recognition error at: '"s=\"'
>
> Syntax error on 41:55 token recognition error at: '"00\"'
>
> Syntax error on 3040:4 token recognition error at: '" \"'
>
> Kind regards,
>
> John.
>
> On Wed, 12 May 2021 at 00:11, antonio <an...@vieiro.net> wrote:
>
>> Thanks John,
>>
>> I'll be adding assorted compiler-specific cases at [1].
>>
>> Kind regards,
>> Antonio
>>
>>
>> [1]
>>
>> https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.antlr4/src/test/resources/org/netbeans/modules/cnd/nextapt/antlr4/c/regressions.c
>>
>> On 11/05/2021 17:41, John Kostaras wrote:
>>> #define CMPLX(__real,__imag) \
>>> _Pragma("clang diagnostic push") \
>>> * _Pragma("clang diagnostic ignored \"-Wcomplex-component-init\"") \*
>>> (double _Complex){(__real),(__imag)} \
>>> _Pragma("clang diagnostic pop")
>>>
>>> line 59 being the highlighted one.
>>>
>>> After pulling, building, re-running: java -Dlexer.print=false -jar
>>> target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include
>>>
>>> I get
>>>
>>> -- /usr/include/complex.h --
>>> Syntax error on 59:12 token recognition error at: '"clang diagnostic
>>> ignored \"'
>>> Please report this error on the dev mailing list, including the line
>> where
>>> the error happeened.%n
>>>
>>>
>>>
>>> On Mon, 10 May 2021 at 23:58, antonio <an...@vieiro.net> wrote:
>>>
>>>> Hi John,
>>>>
>>>> Thanks for the report! It seems we weren't lexing '#error' yet (that's
>>>> probably what is on line 59 in "/usr/include/complex.h", right?
>>>> Otherwise send me the line 59).
>>>>
>>>> The newest version should detect #error properly (and also checks for
>>>> ends of directives, and *.hpp/*.cpp).
>>>>
>>>> Talking about antlr4, it seems the AST is analyzed using the visitor
>>>> pattern (that approach is also used in libclang) but NetBeans <= 8.2
>>>> expects the whole AST in memory, so it seems we have a long refactoring
>>>> road ahead (all this after successfully migrating the CXXParser.g to
>>>> antlr4, which reads >3300 lines long).
>>>>
>>>> In parallel I'm giving libclang [1] a run. It seems quite powerful (it
>>>> provides completion, precompiled headers, reparsing, diagnostics, AST)
>>>> but it may require an extra +30Mb download on Windows (linux users may
>>>> apt-get install libclang, though). More on this in a few days.
>>>>
>>>> Thanks again,
>>>> Antonio
>>>>
>>>> [1]
>>>> https://clang.llvm.org/doxygen/group__CINDEX.html
>>>>
>>>> El 10/5/21 a las 21:39, John Kostaras escribió:
>>>>> +1 cnd.nextapt
>>>>>
>>>>> java -Dlexer.print=false -jar
>>>>> target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar
>> /usr/include
>>>>> ....
>>>>> -- /usr/include/complex.h --
>>>>> Syntax error on 59:12 token recognition error at: '"clang diagnostic
>>>>> ignored \"'
>>>>>
>>>>> Will also take a look. Thanks Antonio. Great initiative.
>>>>>
>>>>> P.S. I guess .hpp/.cpp files are next?
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
>>>> For additional commands, e-mail: dev-help@netbeans.apache.org
>>>>
>>>> For further information about the NetBeans mailing lists, visit:
>>>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>>>
>>>>
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
>> For additional commands, e-mail: dev-help@netbeans.apache.org
>>
>> For further information about the NetBeans mailing lists, visit:
>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>
>>
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [DISCUSS] CND, cnd.antlr & cnd.apt on top of Antlrv4?
Posted by John Kostaras <jk...@gmail.com>.
Hallo Antonio,
so, basically the error is that it cannot handle escape characters, e.g. \".
Here are some tests:
Syntax error on 7613:20 token recognition error at: '" <?xml version=\"'
Syntax error on 3743:20 token recognition error at: '"upd_cmd desc: \"'
Syntax error on 523:3 token recognition error at: '"Error opening file.
PATH: \"'
sprintf (trace_msg,
"Error opening file. PATH: \"%s\" FILE: \"%s\"",
getenv ((char *) PATH), afilename);
Syntax error on 628:37 token recognition error at: '"Cannot get total
number of bytes from message \"'
snprintf(errMsg, MY_ERR_LEN, "Cannot get total number of bytes from
message \"%s\"", sizBuf);
Syntax error on 856:8 token recognition error at: '"<dto-ref ID = \"'
Syntax error on 80:22 token recognition error at: '"Object 1:
Name: \"'
str[0] = (char *) "Object 1: Name: \"OS\"\n";
Syntax error on 29:9 token recognition error at: '"HTML <h3><li><a name=\"'
Syntax error on 388:6 token recognition error at: '"%s DATA
user_id = \"'
Syntax error on 180:12 token recognition error at: '"first line = \"'
Syntax error on 155:8 token recognition error at: '"#include \"'
fputs("#include \"common_types.h\"\n", errorMsg);
Syntax error on 245:3 token recognition error at: '"Host list: \"'
Syntax error on 499:11 token recognition error at: '"\CreateUser (\"'
Syntax error on 67:9 token recognition error at: '"uptime |awk '{print \"'
Syntax error on 51:14 token recognition error at: '"s=\"'
Syntax error on 41:55 token recognition error at: '"00\"'
Syntax error on 3040:4 token recognition error at: '" \"'
Kind regards,
John.
On Wed, 12 May 2021 at 00:11, antonio <an...@vieiro.net> wrote:
> Thanks John,
>
> I'll be adding assorted compiler-specific cases at [1].
>
> Kind regards,
> Antonio
>
>
> [1]
>
> https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.antlr4/src/test/resources/org/netbeans/modules/cnd/nextapt/antlr4/c/regressions.c
>
> On 11/05/2021 17:41, John Kostaras wrote:
> > #define CMPLX(__real,__imag) \
> > _Pragma("clang diagnostic push") \
> > * _Pragma("clang diagnostic ignored \"-Wcomplex-component-init\"") \*
> > (double _Complex){(__real),(__imag)} \
> > _Pragma("clang diagnostic pop")
> >
> > line 59 being the highlighted one.
> >
> > After pulling, building, re-running: java -Dlexer.print=false -jar
> > target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include
> >
> > I get
> >
> > -- /usr/include/complex.h --
> > Syntax error on 59:12 token recognition error at: '"clang diagnostic
> > ignored \"'
> > Please report this error on the dev mailing list, including the line
> where
> > the error happeened.%n
> >
> >
> >
> > On Mon, 10 May 2021 at 23:58, antonio <an...@vieiro.net> wrote:
> >
> >> Hi John,
> >>
> >> Thanks for the report! It seems we weren't lexing '#error' yet (that's
> >> probably what is on line 59 in "/usr/include/complex.h", right?
> >> Otherwise send me the line 59).
> >>
> >> The newest version should detect #error properly (and also checks for
> >> ends of directives, and *.hpp/*.cpp).
> >>
> >> Talking about antlr4, it seems the AST is analyzed using the visitor
> >> pattern (that approach is also used in libclang) but NetBeans <= 8.2
> >> expects the whole AST in memory, so it seems we have a long refactoring
> >> road ahead (all this after successfully migrating the CXXParser.g to
> >> antlr4, which reads >3300 lines long).
> >>
> >> In parallel I'm giving libclang [1] a run. It seems quite powerful (it
> >> provides completion, precompiled headers, reparsing, diagnostics, AST)
> >> but it may require an extra +30Mb download on Windows (linux users may
> >> apt-get install libclang, though). More on this in a few days.
> >>
> >> Thanks again,
> >> Antonio
> >>
> >> [1]
> >> https://clang.llvm.org/doxygen/group__CINDEX.html
> >>
> >> El 10/5/21 a las 21:39, John Kostaras escribió:
> >>> +1 cnd.nextapt
> >>>
> >>> java -Dlexer.print=false -jar
> >>> target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar
> /usr/include
> >>> ....
> >>> -- /usr/include/complex.h --
> >>> Syntax error on 59:12 token recognition error at: '"clang diagnostic
> >>> ignored \"'
> >>>
> >>> Will also take a look. Thanks Antonio. Great initiative.
> >>>
> >>> P.S. I guess .hpp/.cpp files are next?
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> >> For additional commands, e-mail: dev-help@netbeans.apache.org
> >>
> >> For further information about the NetBeans mailing lists, visit:
> >> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
> >>
> >>
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
Re: [DISCUSS] CND, cnd.antlr & cnd.apt on top of Antlrv4?
Posted by antonio <an...@vieiro.net>.
Thanks John,
I'll be adding assorted compiler-specific cases at [1].
Kind regards,
Antonio
[1]
https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.antlr4/src/test/resources/org/netbeans/modules/cnd/nextapt/antlr4/c/regressions.c
On 11/05/2021 17:41, John Kostaras wrote:
> #define CMPLX(__real,__imag) \
> _Pragma("clang diagnostic push") \
> * _Pragma("clang diagnostic ignored \"-Wcomplex-component-init\"") \*
> (double _Complex){(__real),(__imag)} \
> _Pragma("clang diagnostic pop")
>
> line 59 being the highlighted one.
>
> After pulling, building, re-running: java -Dlexer.print=false -jar
> target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include
>
> I get
>
> -- /usr/include/complex.h --
> Syntax error on 59:12 token recognition error at: '"clang diagnostic
> ignored \"'
> Please report this error on the dev mailing list, including the line where
> the error happeened.%n
>
>
>
> On Mon, 10 May 2021 at 23:58, antonio <an...@vieiro.net> wrote:
>
>> Hi John,
>>
>> Thanks for the report! It seems we weren't lexing '#error' yet (that's
>> probably what is on line 59 in "/usr/include/complex.h", right?
>> Otherwise send me the line 59).
>>
>> The newest version should detect #error properly (and also checks for
>> ends of directives, and *.hpp/*.cpp).
>>
>> Talking about antlr4, it seems the AST is analyzed using the visitor
>> pattern (that approach is also used in libclang) but NetBeans <= 8.2
>> expects the whole AST in memory, so it seems we have a long refactoring
>> road ahead (all this after successfully migrating the CXXParser.g to
>> antlr4, which reads >3300 lines long).
>>
>> In parallel I'm giving libclang [1] a run. It seems quite powerful (it
>> provides completion, precompiled headers, reparsing, diagnostics, AST)
>> but it may require an extra +30Mb download on Windows (linux users may
>> apt-get install libclang, though). More on this in a few days.
>>
>> Thanks again,
>> Antonio
>>
>> [1]
>> https://clang.llvm.org/doxygen/group__CINDEX.html
>>
>> El 10/5/21 a las 21:39, John Kostaras escribió:
>>> +1 cnd.nextapt
>>>
>>> java -Dlexer.print=false -jar
>>> target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include
>>> ....
>>> -- /usr/include/complex.h --
>>> Syntax error on 59:12 token recognition error at: '"clang diagnostic
>>> ignored \"'
>>>
>>> Will also take a look. Thanks Antonio. Great initiative.
>>>
>>> P.S. I guess .hpp/.cpp files are next?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
>> For additional commands, e-mail: dev-help@netbeans.apache.org
>>
>> For further information about the NetBeans mailing lists, visit:
>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>
>>
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [DISCUSS] CND, cnd.antlr & cnd.apt on top of Antlrv4?
Posted by John Kostaras <jk...@gmail.com>.
#define CMPLX(__real,__imag) \
_Pragma("clang diagnostic push") \
* _Pragma("clang diagnostic ignored \"-Wcomplex-component-init\"") \*
(double _Complex){(__real),(__imag)} \
_Pragma("clang diagnostic pop")
line 59 being the highlighted one.
After pulling, building, re-running: java -Dlexer.print=false -jar
target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include
I get
-- /usr/include/complex.h --
Syntax error on 59:12 token recognition error at: '"clang diagnostic
ignored \"'
Please report this error on the dev mailing list, including the line where
the error happeened.%n
On Mon, 10 May 2021 at 23:58, antonio <an...@vieiro.net> wrote:
> Hi John,
>
> Thanks for the report! It seems we weren't lexing '#error' yet (that's
> probably what is on line 59 in "/usr/include/complex.h", right?
> Otherwise send me the line 59).
>
> The newest version should detect #error properly (and also checks for
> ends of directives, and *.hpp/*.cpp).
>
> Talking about antlr4, it seems the AST is analyzed using the visitor
> pattern (that approach is also used in libclang) but NetBeans <= 8.2
> expects the whole AST in memory, so it seems we have a long refactoring
> road ahead (all this after successfully migrating the CXXParser.g to
> antlr4, which reads >3300 lines long).
>
> In parallel I'm giving libclang [1] a run. It seems quite powerful (it
> provides completion, precompiled headers, reparsing, diagnostics, AST)
> but it may require an extra +30Mb download on Windows (linux users may
> apt-get install libclang, though). More on this in a few days.
>
> Thanks again,
> Antonio
>
> [1]
> https://clang.llvm.org/doxygen/group__CINDEX.html
>
> El 10/5/21 a las 21:39, John Kostaras escribió:
> > +1 cnd.nextapt
> >
> > java -Dlexer.print=false -jar
> > target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include
> > ....
> > -- /usr/include/complex.h --
> > Syntax error on 59:12 token recognition error at: '"clang diagnostic
> > ignored \"'
> >
> > Will also take a look. Thanks Antonio. Great initiative.
> >
> > P.S. I guess .hpp/.cpp files are next?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
Re: [DISCUSS] CND, cnd.antlr & cnd.apt on top of Antlrv4?
Posted by antonio <an...@vieiro.net>.
Hi John,
Thanks for the report! It seems we weren't lexing '#error' yet (that's
probably what is on line 59 in "/usr/include/complex.h", right?
Otherwise send me the line 59).
The newest version should detect #error properly (and also checks for
ends of directives, and *.hpp/*.cpp).
Talking about antlr4, it seems the AST is analyzed using the visitor
pattern (that approach is also used in libclang) but NetBeans <= 8.2
expects the whole AST in memory, so it seems we have a long refactoring
road ahead (all this after successfully migrating the CXXParser.g to
antlr4, which reads >3300 lines long).
In parallel I'm giving libclang [1] a run. It seems quite powerful (it
provides completion, precompiled headers, reparsing, diagnostics, AST)
but it may require an extra +30Mb download on Windows (linux users may
apt-get install libclang, though). More on this in a few days.
Thanks again,
Antonio
[1]
https://clang.llvm.org/doxygen/group__CINDEX.html
El 10/5/21 a las 21:39, John Kostaras escribió:
> +1 cnd.nextapt
>
> java -Dlexer.print=false -jar
> target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include
> ....
> -- /usr/include/complex.h --
> Syntax error on 59:12 token recognition error at: '"clang diagnostic
> ignored \"'
>
> Will also take a look. Thanks Antonio. Great initiative.
>
> P.S. I guess .hpp/.cpp files are next?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: [DISCUSS] CND, cnd.antlr & cnd.apt on top of Antlrv4?
Posted by John Kostaras <jk...@gmail.com>.
+1 cnd.nextapt
java -Dlexer.print=false -jar
target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include
....
-- /usr/include/complex.h --
Syntax error on 59:12 token recognition error at: '"clang diagnostic
ignored \"'
Will also take a look. Thanks Antonio. Great initiative.
P.S. I guess .hpp/.cpp files are next?
On Thu, 6 May 2021 at 09:52, antonio <an...@vieiro.net> wrote:
> Hi all,
>
> As you may remember from previous threads, the NetBeans 8.2 "cnd.antlr"
> module contained a fork of Antlr 2.7.7 (compiler & runtime) that was not
> donated to the ASF. As a substitute Oracle donated a
> "cnd/libs.cnd.antlr" library wrapper that wrapped the NetBeans 8.2 jar,
> that we cannot use in Apache NetBeans due to licensing problems.
>
> The "cnd.apt" package contains an old "aptlexer.g" (on top of this Antlr
> 2.7.7 runtime) lexer grammar that is responsible for lexing C/C++ and
> Fortran files.
>
> I've done a small experiment (see [1]) transforming this old "aptexpr.g"
> grammar with 1403 lines [2] to a brand new "APTLexer.g4" lexer grammar
> [3] with 480 lines.
>
> This is still on early stages, but it seems the resulting Antlrv4 lexer
> outperforms the old Antlrv2 lexer (we go from ~2600 files/second to
> ~4000 files/second).
>
> I think we can continue the path of removing the old "libs.antlr.cnd"
> module from CND and then refactor "cnd.apt" (and "cnd.asm" et al) on top
> of the new lexer and Antlrv4. The idea is to upgrade all existing
> grammars (including the CXX parser at
>
> https://github.com/apache/netbeans/blob/cnd/cnd/cnd.modelimpl/src/org/netbeans/modules/cnd/modelimpl/parser/CXXParser.g)
>
> to Antlrv4 on top of this new lexer.
>
> Since the "libs.antlr.cnd" API permeates different modules of CND this
> may take quite a long time.
>
> I think we can create a brand new "cnd2" branch in the main repository
> (forked from the "cnd" branch and refreshed with latest 12.4 version)
> and do this CND refactoring on top of Antlrv4 there, right? Any
> objections to this? Or shall I continue this experiment in a repo of
> mine? Any other plans/suggestions to have CND back?
>
> Thanks,
> Antonio
>
>
> P.S.: In order to test the lexer I'd appreciate if you run the new lexer
> (instructions at [1]) on some of your include/source directories and
> report back any syntax errors you may see (the lexer will halt on any
> lexer error), so we can fine-tune any extra lexer rules we may need.
>
>
> [1]
> https://github.com/vieiro/cnd.nextapt
>
> To lex a whole directory of (*.h/*.c files) run the command:
> java -Dlexer.print=false -jar
> target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/include/
>
> To lex a specific file (dumping all tokens to stdout by removing
> -Dlexer.print=false):
> java -jar target/cnd.apt.antlr4-1.0-SNAPSHOT-jar-with-dependencies.jar
> your-file.[ch]
>
> [2]
> https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.nb82/aptlexer.g
>
> [3]
>
> https://github.com/vieiro/cnd.nextapt/blob/master/cnd.apt.antlr4/src/main/antlr4/org/netbeans/modules/cnd/apt/impl/support/generated/APTLexer.g4
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>