You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@freemarker.apache.org by Malte Brunnlieb <m_...@cs.uni-kl.de> on 2015/09/10 11:33:42 UTC
Using FreeMarkers Parser and Lexer
Hi all,
due to my current research, I try to access FreeMarkers Parser and/or
Lexer appropriately, but it seems, that you have already successfully
hidden the most necessary classes to do so.
The only thing, I am currently able to do is to get the parsed element
root. Nevertheless, also this API seems to be deprecated.
I also tried to work directly with the FTL.jj defintion, but you adapted
the API of the Parser and Lexer Exceptions I cannot simply reuse from
your library.
I know that my issue is a very special and advanced use case, but do you
see any possibility to work with the latest FreeMarker Lexer/Parser by
simple library reuse?
Best Regards,
Malte
Re: AW: AW: Using FreeMarkers Parser and Lexer
Posted by Daniel Dekany <dd...@freemail.hu>.
Sunday, September 13, 2015, 1:15:05 PM, Malte Brunnlieb wrote:
> Hi Daniel,
>
> I try to transform the FreeMarker javaCC grammar to an ANTLR4 grammar.
> I did not get the sematics of this lexer state syntax:
>
> <FM_EXPRESSION, IN_PAREN, NAMED_PARAMETER_EXPRESSION> SKIP :
> {
> ...
> }
>
> What does this mean?
The stuff inside <...> means that the patterns inside the {...} are
only used in the lexical states listed there (logical "or").
> Does this match the current stack of lexer states? Does JavaCC
> manage lexer states in a stack anyhow?
At any moment the lexer is in a single state which is identified with
a name (initially DEFAULT), and that's only what it matches. There's
no stack as far as I know. The regular expressions switch between
states directly.
> I googled a lot but could not find this syntax in the web.
> If so, how to push/pop something from the lexer stack in javaCC?
One could maintain a such stack manually, since you can have custom
lexer state fields. It doesn't solve matching against stack content
efficiently, but at least you can push and pop, with your own methods.
> I just could find the swithTo(...) trigger.
>
> Best Regards,
> Malte
>
> -----Ursprüngliche Nachricht-----
> Von: Daniel Dekany [mailto:ddekany@freemail.hu]
> Gesendet: Samstag, 12. September 2015 19:40
> An: Malte Brunnlieb
> Cc: dev@freemarker.incubator.apache.org
> Betreff: Re: AW: Using FreeMarkers Parser and Lexer
>
> The solution for FTL would be quite similar to Javaparser: there should be
> *another* project that provides an FTL parser, but that's not (or not
> exactly) the same parser than that FTL uses internally, similarly to as
> Javaparser is not the same parser that javac uses internally. Maybe the best
> way of making a such parser is copying and then modifying the current
> parser, and ideally, that project is also helped by people who work on
> FreeMarker itself, so that new language features will get in, and test cases
> for new FTL features are provided. But someone had to take this quite big
> task on himself, with longer term maintenance ideally.
>
> Also, I think you will be better of by copying the current stuff into
> another package (maybe under a liberal license), than accessing things with
> reflection and such that was told to be internal. In the first case you
> might won't support the latest language features after a FreeMarker update,
> but at least your code won't die with ClassNotFound and NoSuchMethod after
> the same FreeMarker update.
>
> (To understand my standpoint more, my opinion is that one day, the current
> parser will have to be heavily reworked, together with some other internals.
> The goal of that is making the language more extensible and making templates
> more introspectable with public API-s.
> Another thing worths knowing is that keeping backward compatibility causes a
> lot of hardship for FM, partly because of API-s negligently exposed in the
> old days. So I have become very careful with undertaking new backward
> compatibility obligations.)
>
> --
> Thanks,
> Daniel Dekany
>
>
> Saturday, September 12, 2015, 3:24:08 PM, Malte Brunnlieb wrote:
>
>> Hi,
>>
>> I am currently trying to work with the Lexer/Parser of FreeMarker as well
> as
>> the Lexer/Parser of the target language to support syntax-safe generation.
>> Thus, it would be nice to just use the FreeMarker library to parse
>> FreeMarker templates as it is also possible with the javaparser I am
> using.
>>
>> I respect the idea of only providing APIs, which will be backward
> compatible
>> in future. Nevertheless, The TokenManager as well as the Parser do follow
> a
>> commonly used API Pattern, which should be quite stable and could be made
>> accessible. I do not claim to support these APIs actively by providing any
>> documentation. It just makes life easier for those, who want to provide
>> further tool support for FreeMarker template editors as well as further
>> language engineers trying to use or adapt FreeMarker as a meta-language
> for
>> any (possibly research) intention. Providing the lexer and parser with
> your
>> deployment for free eventually would establish FreeMarker even more due to
>> more potential tool support.
>>
>> Currently, I somehow managed to work with the non-public TokenManager and
>> Parser API by hacking accessibility via reflection.
>> I know this is not a stable approach, but I am fine with it to just get
> the
>> lexer token stream and the parser token stream to work on further ideas.
>>
>> Best Regards,
>> Malte
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Daniel Dekany [mailto:ddekany@freemail.hu]
>> Gesendet: Freitag, 11. September 2015 17:53
>> An: Malte Brunnlieb
>> Betreff: Re: Using FreeMarkers Parser and Lexer
>>
>> That the related API-s are not public or are public but marked as
>> internal/deprecated is because FM doesn't provide backward compatibility
>> there. So you might as well copy them into your package and change the
>> visibilities, etc.
>>
>> What exactly do you want to achieve?
>>
>>
>> Thursday, September 10, 2015, 11:33:42 AM, Malte Brunnlieb wrote:
>>
>>> Hi all,
>>>
>>> due to my current research, I try to access FreeMarkers Parser and/or
>>> Lexer appropriately, but it seems, that you have already successfully
>>> hidden the most necessary classes to do so.
>>> The only thing, I am currently able to do is to get the parsed element
>>> root. Nevertheless, also this API seems to be deprecated.
>>>
>>> I also tried to work directly with the FTL.jj defintion, but you
>>> adapted the API of the Parser and Lexer Exceptions I cannot simply
>>> reuse from your library.
>>>
>>> I know that my issue is a very special and advanced use case, but do
>>> you see any possibility to work with the latest FreeMarker
>>> Lexer/Parser by simple library reuse?
>>>
>>> Best Regards,
>>> Malte
>>>
>>>
>>
>> --
>> Thanks,
>> Daniel Dekany
>
>
>
--
Thanks,
Daniel Dekany
AW: AW: Using FreeMarkers Parser and Lexer
Posted by Malte Brunnlieb <m_...@cs.uni-kl.de>.
Hi Daniel,
I try to transform the FreeMarker javaCC grammar to an ANTLR4 grammar.
I did not get the sematics of this lexer state syntax:
<FM_EXPRESSION, IN_PAREN, NAMED_PARAMETER_EXPRESSION> SKIP :
{
...
}
What does this mean? Does this match the current stack of lexer states?
Does JavaCC manage lexer states in a stack anyhow? I googled a lot but could
not find this syntax in the web.
If so, how to push/pop something from the lexer stack in javaCC? I just
could find the swithTo(...) trigger.
Best Regards,
Malte
-----Ursprüngliche Nachricht-----
Von: Daniel Dekany [mailto:ddekany@freemail.hu]
Gesendet: Samstag, 12. September 2015 19:40
An: Malte Brunnlieb
Cc: dev@freemarker.incubator.apache.org
Betreff: Re: AW: Using FreeMarkers Parser and Lexer
The solution for FTL would be quite similar to Javaparser: there should be
*another* project that provides an FTL parser, but that's not (or not
exactly) the same parser than that FTL uses internally, similarly to as
Javaparser is not the same parser that javac uses internally. Maybe the best
way of making a such parser is copying and then modifying the current
parser, and ideally, that project is also helped by people who work on
FreeMarker itself, so that new language features will get in, and test cases
for new FTL features are provided. But someone had to take this quite big
task on himself, with longer term maintenance ideally.
Also, I think you will be better of by copying the current stuff into
another package (maybe under a liberal license), than accessing things with
reflection and such that was told to be internal. In the first case you
might won't support the latest language features after a FreeMarker update,
but at least your code won't die with ClassNotFound and NoSuchMethod after
the same FreeMarker update.
(To understand my standpoint more, my opinion is that one day, the current
parser will have to be heavily reworked, together with some other internals.
The goal of that is making the language more extensible and making templates
more introspectable with public API-s.
Another thing worths knowing is that keeping backward compatibility causes a
lot of hardship for FM, partly because of API-s negligently exposed in the
old days. So I have become very careful with undertaking new backward
compatibility obligations.)
--
Thanks,
Daniel Dekany
Saturday, September 12, 2015, 3:24:08 PM, Malte Brunnlieb wrote:
> Hi,
>
> I am currently trying to work with the Lexer/Parser of FreeMarker as well
as
> the Lexer/Parser of the target language to support syntax-safe generation.
> Thus, it would be nice to just use the FreeMarker library to parse
> FreeMarker templates as it is also possible with the javaparser I am
using.
>
> I respect the idea of only providing APIs, which will be backward
compatible
> in future. Nevertheless, The TokenManager as well as the Parser do follow
a
> commonly used API Pattern, which should be quite stable and could be made
> accessible. I do not claim to support these APIs actively by providing any
> documentation. It just makes life easier for those, who want to provide
> further tool support for FreeMarker template editors as well as further
> language engineers trying to use or adapt FreeMarker as a meta-language
for
> any (possibly research) intention. Providing the lexer and parser with
your
> deployment for free eventually would establish FreeMarker even more due to
> more potential tool support.
>
> Currently, I somehow managed to work with the non-public TokenManager and
> Parser API by hacking accessibility via reflection.
> I know this is not a stable approach, but I am fine with it to just get
the
> lexer token stream and the parser token stream to work on further ideas.
>
> Best Regards,
> Malte
>
> -----Ursprüngliche Nachricht-----
> Von: Daniel Dekany [mailto:ddekany@freemail.hu]
> Gesendet: Freitag, 11. September 2015 17:53
> An: Malte Brunnlieb
> Betreff: Re: Using FreeMarkers Parser and Lexer
>
> That the related API-s are not public or are public but marked as
> internal/deprecated is because FM doesn't provide backward compatibility
> there. So you might as well copy them into your package and change the
> visibilities, etc.
>
> What exactly do you want to achieve?
>
>
> Thursday, September 10, 2015, 11:33:42 AM, Malte Brunnlieb wrote:
>
>> Hi all,
>>
>> due to my current research, I try to access FreeMarkers Parser and/or
>> Lexer appropriately, but it seems, that you have already successfully
>> hidden the most necessary classes to do so.
>> The only thing, I am currently able to do is to get the parsed element
>> root. Nevertheless, also this API seems to be deprecated.
>>
>> I also tried to work directly with the FTL.jj defintion, but you
>> adapted the API of the Parser and Lexer Exceptions I cannot simply
>> reuse from your library.
>>
>> I know that my issue is a very special and advanced use case, but do
>> you see any possibility to work with the latest FreeMarker
>> Lexer/Parser by simple library reuse?
>>
>> Best Regards,
>> Malte
>>
>>
>
> --
> Thanks,
> Daniel Dekany
Re: AW: Using FreeMarkers Parser and Lexer
Posted by Daniel Dekany <dd...@freemail.hu>.
The solution for FTL would be quite similar to Javaparser: there
should be *another* project that provides an FTL parser, but that's
not (or not exactly) the same parser than that FTL uses internally,
similarly to as Javaparser is not the same parser that javac uses
internally. Maybe the best way of making a such parser is copying and
then modifying the current parser, and ideally, that project is also
helped by people who work on FreeMarker itself, so that new language
features will get in, and test cases for new FTL features are
provided. But someone had to take this quite big task on himself, with
longer term maintenance ideally.
Also, I think you will be better of by copying the current stuff into
another package (maybe under a liberal license), than accessing things
with reflection and such that was told to be internal. In the first
case you might won't support the latest language features after a
FreeMarker update, but at least your code won't die with ClassNotFound
and NoSuchMethod after the same FreeMarker update.
(To understand my standpoint more, my opinion is that one day, the
current parser will have to be heavily reworked, together with some
other internals. The goal of that is making the language more
extensible and making templates more introspectable with public API-s.
Another thing worths knowing is that keeping backward compatibility
causes a lot of hardship for FM, partly because of API-s negligently
exposed in the old days. So I have become very careful with
undertaking new backward compatibility obligations.)
--
Thanks,
Daniel Dekany
Saturday, September 12, 2015, 3:24:08 PM, Malte Brunnlieb wrote:
> Hi,
>
> I am currently trying to work with the Lexer/Parser of FreeMarker as well as
> the Lexer/Parser of the target language to support syntax-safe generation.
> Thus, it would be nice to just use the FreeMarker library to parse
> FreeMarker templates as it is also possible with the javaparser I am using.
>
> I respect the idea of only providing APIs, which will be backward compatible
> in future. Nevertheless, The TokenManager as well as the Parser do follow a
> commonly used API Pattern, which should be quite stable and could be made
> accessible. I do not claim to support these APIs actively by providing any
> documentation. It just makes life easier for those, who want to provide
> further tool support for FreeMarker template editors as well as further
> language engineers trying to use or adapt FreeMarker as a meta-language for
> any (possibly research) intention. Providing the lexer and parser with your
> deployment for free eventually would establish FreeMarker even more due to
> more potential tool support.
>
> Currently, I somehow managed to work with the non-public TokenManager and
> Parser API by hacking accessibility via reflection.
> I know this is not a stable approach, but I am fine with it to just get the
> lexer token stream and the parser token stream to work on further ideas.
>
> Best Regards,
> Malte
>
> -----Ursprüngliche Nachricht-----
> Von: Daniel Dekany [mailto:ddekany@freemail.hu]
> Gesendet: Freitag, 11. September 2015 17:53
> An: Malte Brunnlieb
> Betreff: Re: Using FreeMarkers Parser and Lexer
>
> That the related API-s are not public or are public but marked as
> internal/deprecated is because FM doesn't provide backward compatibility
> there. So you might as well copy them into your package and change the
> visibilities, etc.
>
> What exactly do you want to achieve?
>
>
> Thursday, September 10, 2015, 11:33:42 AM, Malte Brunnlieb wrote:
>
>> Hi all,
>>
>> due to my current research, I try to access FreeMarkers Parser and/or
>> Lexer appropriately, but it seems, that you have already successfully
>> hidden the most necessary classes to do so.
>> The only thing, I am currently able to do is to get the parsed element
>> root. Nevertheless, also this API seems to be deprecated.
>>
>> I also tried to work directly with the FTL.jj defintion, but you
>> adapted the API of the Parser and Lexer Exceptions I cannot simply
>> reuse from your library.
>>
>> I know that my issue is a very special and advanced use case, but do
>> you see any possibility to work with the latest FreeMarker
>> Lexer/Parser by simple library reuse?
>>
>> Best Regards,
>> Malte
>>
>>
>
> --
> Thanks,
> Daniel Dekany
AW: Using FreeMarkers Parser and Lexer
Posted by Malte Brunnlieb <ma...@brunnlieb.net>.
Hi,
I am currently trying to work with the Lexer/Parser of FreeMarker as well as
the Lexer/Parser of the target language to support syntax-safe generation.
Thus, it would be nice to just use the FreeMarker library to parse
FreeMarker templates as it is also possible with the javaparser I am using.
I respect the idea of only providing APIs, which will be backward compatible
in future. Nevertheless, The TokenManager as well as the Parser do follow a
commonly used API Pattern, which should be quite stable and could be made
accessible. I do not claim to support these APIs actively by providing any
documentation. It just makes life easier for those, who want to provide
further tool support for FreeMarker template editors as well as further
language engineers trying to use or adapt FreeMarker as a meta-language for
any (possibly research) intention. Providing the lexer and parser with your
deployment for free eventually would establish FreeMarker even more due to
more potential tool support.
Currently, I somehow managed to work with the non-public TokenManager and
Parser API by hacking accessibility via reflection.
I know this is not a stable approach, but I am fine with it to just get the
lexer token stream and the parser token stream to work on further ideas.
Best Regards,
Malte
-----Ursprüngliche Nachricht-----
Von: Daniel Dekany [mailto:ddekany@freemail.hu]
Gesendet: Freitag, 11. September 2015 17:53
An: Malte Brunnlieb
Betreff: Re: Using FreeMarkers Parser and Lexer
That the related API-s are not public or are public but marked as
internal/deprecated is because FM doesn't provide backward compatibility
there. So you might as well copy them into your package and change the
visibilities, etc.
What exactly do you want to achieve?
Thursday, September 10, 2015, 11:33:42 AM, Malte Brunnlieb wrote:
> Hi all,
>
> due to my current research, I try to access FreeMarkers Parser and/or
> Lexer appropriately, but it seems, that you have already successfully
> hidden the most necessary classes to do so.
> The only thing, I am currently able to do is to get the parsed element
> root. Nevertheless, also this API seems to be deprecated.
>
> I also tried to work directly with the FTL.jj defintion, but you
> adapted the API of the Parser and Lexer Exceptions I cannot simply
> reuse from your library.
>
> I know that my issue is a very special and advanced use case, but do
> you see any possibility to work with the latest FreeMarker
> Lexer/Parser by simple library reuse?
>
> Best Regards,
> Malte
>
>
--
Thanks,
Daniel Dekany
Re: Using FreeMarkers Parser and Lexer
Posted by Daniel Dekany <dd...@freemail.hu>.
That the related API-s are not public or are public but marked as
internal/deprecated is because FM doesn't provide backward
compatibility there. So you might as well copy them into your package
and change the visibilities, etc.
What exactly do you want to achieve?
Thursday, September 10, 2015, 11:33:42 AM, Malte Brunnlieb wrote:
> Hi all,
>
> due to my current research, I try to access FreeMarkers Parser and/or
> Lexer appropriately, but it seems, that you have already successfully
> hidden the most necessary classes to do so.
> The only thing, I am currently able to do is to get the parsed element
> root. Nevertheless, also this API seems to be deprecated.
>
> I also tried to work directly with the FTL.jj defintion, but you adapted
> the API of the Parser and Lexer Exceptions I cannot simply reuse from
> your library.
>
> I know that my issue is a very special and advanced use case, but do you
> see any possibility to work with the latest FreeMarker Lexer/Parser by
> simple library reuse?
>
> Best Regards,
> Malte
>
>
--
Thanks,
Daniel Dekany