My FreeMarker 3 wishlist

[Emmeran Seehuber <ro...@rototor.de>]

Hi all,

I am using FreeMarker since 6 years now. Its working very well for my web-frontend, email-templating and (in combination with Fyling Saucer / OpenHtmlToPDF https://github.com/danfickle/openhtmltopdf <https://github.com/danfickle/openhtmltopdf>) my PDF reporting needs.

It`s cool that FreeMarker is actively develop, even if the current version already has more or less everything I need. The minor releases you did the last years had all nice little feature additions, which were not required, but useful never the less.

For FreeMarker 3 I would have the following requirement: 

Feel free to throw around the Java-API/implementation however you like. If I get a ton of compile errors and just have to follow a migration guide, then I am happy to do so. That would not be a problem.

But please try to stay as compatible with the existing templates as possible. If you really have/want to remove a builtin or system global because its deprecated and/or limiting the evolution of FreeMarker then ensure that the templates using the old constructs throw an error at parse time. As I always preload all templates when starting my servers in production mode that would allow me to catch all migration issues. If those removed bultins would only throw an error at runtime this would be a show stopper for me. I would not be able to migration to FreeMarker 3. And I think that would block the migration for other projects too. So please be careful with removing things from the template language.

The following features would be great in FreeMarker 3:

- An official way to define variable types in templates. IntellJ Ultimate fully supports FreeMarker and allows to define explicit types for variables and macro parameters (https://www.jetbrains.com/help/idea/2016.3/template-languages-velocity-and-freemarker.html <https://www.jetbrains.com/help/idea/2016.3/template-languages-velocity-and-freemarker.html>). You can define them globally for your project and per file / macro. I.e

To define a global root model:
[#-- @ftlvariable name="" type=„xy.myapplcation.model.PageRootFacade"  --]

To define a single variable / macro parameter:
[#-- @ftlvariable name="confluenceArtikel" type="xy.myapplcation.model.ConfluenceArticleFacade" —]

Using this comments IntellJ is able to provide full content assist, javadocs and even navigation to the Java model object sources. That way FreeMarker gets „semistatic“ typed in the IDE and non existing property/method names get flagged by IntelliJ. 

Some official supported statement to define this types would be great, e.g.

[#variable name="confluenceArtikel" type="xy.myapplcation.model..ConfluenceArticleFacade“]
or
<#variable name="confluenceArtikel" type="xy.myapplcation.model..ConfluenceArticleFacade“>

depending on your bracket style. 

- Separate the parser AST from the ExecutionTree. Currently the parser AST is also the ExecutionTree and gets executed in place. This is fast to build, simple and works well. But the way it is now blocks many optimizations. Ideally the pass to convert the parser AST to the ExecutionTree  could be configured. A „simple“ and fast mode as default, which just converts the parser AST to an ExecuteTree like it is now. But it would be possible to write a optimizing pass which uses the type information from the <#variable> declarations and create bytecode which tries to do as many things as static typed as possible. Such an optimizing pass / mode is of course optional and could be implemented later and even require the newest Java version. If delivered as optional maven dependency it could also use tools like ByteBuddy.

- If the parser AST would be an official API it would allow IDEs to use it to parse the structure and type information from error free templates and build an index. Even if it would just be an "inofficial, can change with every release“-API I think it could help other IDEs a lot. (IntellJ IDEA has/will always implement its own parser, but Eclipse or NetBeans could use that).

- An parser AST would also allow to walk the tree to extract method calls. Currently I use reflection (+ setAccessible(true)) to extract all method calls to a „magic“ method „_“, which I use for gettext style translations. E.g. from a template like: 

<buttton>${_(„My shiny button“)}</button>

the text „My shiny button“ is extracted and put it into a .po file for future translation. It works, but is a hack, and sometimes needs fixes when upgrading FreeMarker as internal structures/variables change. An official way to do such things would be nice.

To be honest, I don`t think that I will have time to implement any of that for FreeMarker 3. So this are just suggestions. But maybe some of this, especially building an optimized ExecutionTree, could attract new people.

Thanks for this great project and keep up the good work!

cu,
  Emmy

--
Emmeran Seehuber
Dipl. Inf. (FH)
Schrannenstraße 8
86150 Augsburg
USt-IdNr.: DE266070804

Re: My FreeMarker 3 wishlist

[Daniel Dekany <dd...@freemail.hu>]

Sunday, January 8, 2017, 1:36:36 PM, Emmeran Seehuber wrote:

> Hi all,
>
> I am using FreeMarker since 6 years now. Its working very well for
> my web-frontend, email-templating and (in combination with Fyling
> Saucer / OpenHtmlToPDF https://github.com/danfickle/openhtmltopdf
> <https://github.com/danfickle/openhtmltopdf>) my PDF reporting needs.
>
> It`s cool that FreeMarker is actively develop, even if the current
> version already has more or less everything I need. The minor
> releases you did the last years had all nice little feature
> additions, which were not required, but useful never the less.
>
> For FreeMarker 3 I would have the following requirement: 
>
> Feel free to throw around the Java-API/implementation however you
> like. If I get a ton of compile errors and just have to follow a
> migration guide, then I am happy to do so. That would not be a problem.
>
> But please try to stay as compatible with the existing templates as
> possible.

Thing is, there's no version planned where we break API backward
compatibility, but not template language language backward
compatibility. FM2 keeps both. FM3 keeps neither. Because cleaning the
template language is much more important than cleaning up the API, and
if you change the template language anyway, changing the API is not a
big deal next to that.

I don't expect many to migrate existing templates from FM2 to FM3. We
could (and maybe will) provide a tool that *helps* translating FM2
templates to the FM3 syntax, but it's impossible to deal reliably with
the semantical differences (which is also affected by ObjectWrapper
behavior), so, yeah, it's still expensive to do. So, FM3 is mostly for
new project. Better said, old projects can start using FM3 templates
next to the existing FM2 templates (maybe slowly replacing the FM2
ones), because the two version uses different Java package and so they
don't interfere with each other.

This is my stand point anyway. I do maintenance at FM for a good while
(and the original authors were moved on long ago), and I can tell that
with that 14 years of legacy, improving FM 2 has become more and more
difficult if you keep backward compatibility, especially on the
template level. So, we got a slow paced FM 2 project, which is not
very sexy if you want to catch contributors, and which has a language
that's too far from idea IMO. I hope that ultimately FM2 will be taken
over by some fresh template engine, and it will gradually fall out of
use (actually, that has already happened on some more specialized
areas; see Thymeleaf). FM3 is basically a such fresh template engine,
which branches out of FM2, and tries to fix the things that turned out
to be problems during the years. That simply not feasible to do in
FM2. And again, FM2 is here to stay, until something, maybe FM3, takes
over. It can be several years.

> If you really have/want to remove a builtin or system
> global because its deprecated and/or limiting the evolution of
> FreeMarker then ensure that the templates using the old constructs
> throw an error at parse time. As I always preload all templates when
> starting my servers in production mode that would allow me to catch
> all migration issues. If those removed bultins would only throw an
> error at runtime this would be a show stopper for me. I would not be
> able to migration to FreeMarker 3. And I think that would block the
> migration for other projects too. So please be careful with removing
> things from the template language.

> The following features would be great in FreeMarker 3:
>
> - An official way to define variable types in templates. IntellJ
> Ultimate fully supports FreeMarker and allows to define explicit
> types for variables and macro parameters
> (https://www.jetbrains.com/help/idea/2016.3/template-languages-velocity-and-freemarker.html
> <https://www.jetbrains.com/help/idea/2016.3/template-languages-velocity-and-freemarker.html>).
> You can define them globally for your project and per file / macro. I.e
>
> To define a global root model:
> [#-- @ftlvariable name=""
> type=\u201exy.myapplcation.model.PageRootFacade"  --]
>
> To define a single variable / macro parameter:
> [#-- @ftlvariable name="confluenceArtikel"
> type="xy.myapplcation.model.ConfluenceArticleFacade" \u2014]

Yes, that something that's often requested. It should just be done in
FM2 (and of course then in FM2 too), as it's not a breaking change. I
find that syntax a bit to verbose though. We could invent our own,
though then someone should update the IntelliJ plugin. However, it
wouldn't be too hard, as then Template would have a some method like
getDataModelDeclarations() or such.

> Using this comments IntellJ is able to provide full content assist,
> javadocs and even navigation to the Java model object sources. That
> way FreeMarker gets \u201esemistatic\u201c typed in the IDE and non existing
> property/method names get flagged by IntelliJ. 
>
> Some official supported statement to define this types would be great, e.g.
>
> [#variable name="confluenceArtikel"
> type="xy.myapplcation.model..ConfluenceArticleFacade\u201c]
> or
> <#variable name="confluenceArtikel"
> type="xy.myapplcation.model..ConfluenceArticleFacade\u201c>

Or

  <#variableType
      confluenceArtikel="xy.myapplcation.model.ConfluenceArticleFacade"
      sinethingelse="foo.Bar"
  >

> depending on your bracket style. 
>
> - Separate the parser AST from the ExecutionTree. Currently the
> parser AST is also the ExecutionTree and gets executed in place.
> This is fast to build, simple and works well. But the way it is now
> blocks many optimizations. Ideally the pass to convert the parser
> AST to the ExecutionTree  could be configured. A \u201esimple\u201c and fast
> mode as default, which just converts the parser AST to an
> ExecuteTree like it is now. But it would be possible to write a
> optimizing pass which uses the type information from the <#variable>
> declarations and create bytecode which tries to do as many things as
> static typed as possible. Such an optimizing pass / mode is of
> course optional and could be implemented later and even require the
> newest Java version. If delivered as optional maven dependency it
> could also use tools like ByteBuddy.

I think you run into a FM2 limitation there (which is something I will
want to discuss, because removing it is a brutal change). We are using
TemplateModel-s made by ObjectWrapper-s to access the objects. So if
you have `foo.bar`, you as the template executor only know that `foo`
must be a TemplateHashModel whose get("bar") method must be called.
You don't know what does it do internally (Map.get("bar"),
FooBean.getBar(), etc.) You need some kind of MOP implementation that
tells you depending on the operand class (not the instance!) *how* to
execute the operation (i.e., what Java method to call) instead of
executing it itself, and then you have a chance to inline that.

> - If the parser AST would be an official API it would allow IDEs to
> use it to parse the structure and type information from error free
> templates and build an index. Even if it would just be an
> "inofficial, can change with every release\u201c-API I think it could
> help other IDEs a lot. (IntellJ IDEA has/will always implement its
> own parser, but Eclipse or NetBeans could use that).
>
> - An parser AST would also allow to walk the tree to extract method
> calls. Currently I use reflection (+ setAccessible(true)) to extract
> all method calls to a \u201emagic\u201c method \u201e_\u201c, which I use for gettext
> style translations. E.g. from a template like: 
>
> <buttton>${_(\u201eMy shiny button\u201c)}</button>
>
> the text \u201eMy shiny button\u201c is extracted and put it into a .po file
> for future translation. It works, but is a hack, and sometimes needs
> fixes when upgrading FreeMarker as internal structures/variables
> change. An official way to do such things would be nice.

One goal of FM3 is to make the language more "regular", which among
others means that it's easier to expose a tree that's useful for IDE-s
or other kind of template content analysis.

The FM2 AST is quite messy... it's for execution as you said. It could
get nice facade, it was just never done.

> To be honest, I don`t think that I will have time to implement any
> of that for FreeMarker 3. So this are just suggestions. But maybe
> some of this, especially building an optimized ExecutionTree, could attract new people.
>
> Thanks for this great project and keep up the good work!
>
> cu,
>   Emmy
>
> --
> Emmeran Seehuber
> Dipl. Inf. (FH)
> Schrannenstra�e 8
> 86150 Augsburg
> USt-IdNr.: DE266070804

-- 
Thanks,
 Daniel Dekany