[FM3] Package structure changes

[Daniel Dekany <dd...@freemail.hu>]

A bit of prelude first... This is about FreeMarker 3 development (FM3
from now on). If life and the community allows it, there will be tons
of smaller and quite a few bigger changes in FM3. I intend to drop
together a mail like this for most "change sets", so we can see if
more discussion (and voting in worse case) is needed. Generally, I
will wait for answers for ~72 hours, then even if there was no answer,
I will assume that it's accepted (because anything can be undone later
after all, it's just extra work).

So, to the subject. Please tell your opinion of the points below.


(1) Using org.apache.freemarker (o.a.f) as top level package

As it was discussed long ago, FreeMarker 3 (FM3 from now on) will use
a separate parent package from FM2, org.apache.freemarker (o.a.f from
now on). (I also had idea that it could be org.apache.freemarker.v3,
but org.apache.freemarker is surely look cleaner.) The main point of
that is that hence FM2 and FM3 can be used together in the same "class
path", which is important because the main point of FM3 is to get rid
of the backward compatibility baggage.

This was discussed once, but if anyone has anything add, do it.


(2) Unify freemarker.template and freemarker.core, under o.a.f

In FM2 we have a separate freemarler.template package, which contains
the most commonly used classes (Configuration, Template, TemplateModel
classes), and there's a freemarker.core package which contains some
less often used ones (such as Environment), and lots of classes that
are public but was marked as internal (AST and such), and of course
non-public internals too. The root "freemarker" package itself is unused.

There were two recurring problems with this in FM2:

- What is most- and least commonly used is a blurry line, so it's not
  very helpful classification for the API users. Especially if we hide
  the internal but sadly public classes (and we do that on the JavaDoc
  level even in FM2), the concern (topic) of the two packages isn't
  clear. Both deal with core engine stuff really. (And
  freemarker.template is just a misnomer... at least it never made
  sense to me. Why "template"?)

- The freemarker.template package and the freemarker.core package needs to
  cross-reference each other a lot (as their concerns aren't well
  separated from engineering point of view either). As Java's
  visibility rules are rather simplistic (and the Java 9 module system
  won't be common place anytime soon), this forces us to make things
  public that shouldn't be that, or to do hacks with "_XxxApi"
  classes.

Therefor I believe we should just get rid of the freemarker.template
and freemarker.core spearation, and move their contents into o.a.f.
(One the same time we should move out some classes from it, but see
later.)


(3) Move TemplateModel interfaces and standard implementations from
o.a.f to o.a.f.model

Template models is a topic that's easy to understand for the users
(mentally and technically separate from the other concerns), so we
could decrease the surface area of the o.f package API with this.

As of the TemplateModel implementations, let's say for now that we
move them over from o.a.f too. Though there will be some more moving
around in that department, together with BeansWrapper and its friends,
but that's for another day.


(4) Rename freemarker.cache to o.f.templateloader

Because, it was misnomer. Template loading is the main topic of the
package (caching of the TemplateLoader-ed templates is part of it).

-- 
Thanks,
 Daniel Dekany

Re: [FM3] To TemplateModel or not, also ?foo vs standard Java API methods (Was: [FM3] Package structure changes)

[Daniel Dekany <dd...@freemail.hu>]

I have changed the subject to avoid derailing the original thread. See
my comments below...


Wednesday, January 11, 2017, 7:46:22 PM, David E Jones wrote:

> On Wed, 2017-01-11 at 11:55 +0100, Daniel Dekany wrote:
>> 
>> (3) Move TemplateModel interfaces and standard implementations from
>> o.a.f to o.a.f.model
>> 
>> Template models is a topic that's easy to understand for the users
>> (mentally and technically separate from the other concerns), so we
>> could decrease the surface area of the o.f package API with this.
>> 
>> As of the TemplateModel implementations, let's say for now that we
>> move them over from o.a.f too. Though there will be some more moving
>> around in that department, together with BeansWrapper and its friends,
>> but that's for another day.
>
> This may also be a topic for another day, but IIRC there was some
> discussion about getting rid of many of these interfaces in FM3.

Yes, and some day I meant to bring that up too. But that's high risk
thing, as it's a quite brutal change. The code base (and indeed the
mental model) is full of playing around with TemplateModels. So for
now (in the original thread), I just want to move the "model" and
object wrapper stuff somewhere into it's own package.

> I like the idea of more generally using standard interfaces such as
> java.util.List and java.util.Map, and even more
> generally avoiding the object wrapping that FreeMarker currently does.

I prefer the idea of using some kind of "navigator" (or MOP
implementator or whatever we call it). That means that I don't care if
it's a List or array or what, as far as the Navigator (coming from the
Configuration) can tell me how to iterate through it (for #list,
?join, etc.), or how to get the nth item of it (for val[n]). Then
object wrapping is avoided, but, this also means that the decision of
the Navigator (how to do this and that with the particular object) is
not memorized, or at least not as directly as with object wrapping.
But a Navigator can decide quite quickly if it only cares about the
class of the object, and so it can look up the implementation of the
requested operation in a cache keyed with the class name. Not to
mention the remote possibility of using "indy" (invoke dynamic),
because unlike with object wrapping, you decide what to do based on
class, not on instance, so you can use some custom call place that has
the cases for the most frequent classes hard wired (and so
JIT-optimized).

> This may complicate some of the internal introspection code and
> possibly even make certain things slower, but overall
> avoiding object wrapping helps a lot with memory and garbage
> collection overhead and makes the template engine more
> flexible.
>
> On the topic of using more standard Java interfaces and API: many
> built-ins can be eliminated in favor of existing API
> methods, though perhaps some current FreeMarker users prefer the
> ?foo syntax of built-ins versus the .foo() syntax of
> API calls.

There are some problems with relying on the Java API-s of the Java
objects of the data-model.

You might not want to expose all methods to the template authors, but
with an ever expanding set of methods, how to decide which one is OK
to expose to a template?

Also, you might want to use multiple classes (like List VS array is
the most common one) behind the scenes but don't want the templates to
be affected.

Also, many functions are missing from the standard API-s that you want
in templates (like the String API is very minimalist), and it's
confusing for some users if in one case you have to look up things in
some standard Java API, and in another case in the FreeMarker Manual.
Actually, we have this problem with the most basic things too, like
string `?length`; the Java API provides String.length() too. (And if I
want to allow str.length() in templates, then note that now you can
use `.` with something that's not a map-like-value, unlike in FM2.
Solvable, just a note, for now.)

Also there's the problem with Java versions. If Java 8 provides
something that's often needed in templates, and hence we don't add a
built-in for it, the users on Java 5-7 are out of luck, despite that
that 8 isn't the required minimum version. If we add it as a built-in,
then sooner or later (as Java 8+ becomes the norm) it will become to a
duplicate, which can't be removed for backward compatibility.

> For any new built-ins, like the date/time ones recently discussed,
> they could at least be inspired by the more standard Java APIs like
> the java.time classes as opposed to the Joda Time ones.

To be inspired by them is certainly a good idea.

> Joda Time is a great little package, but for stuff I've been doing
> lately it has been very nice to migrate to the
> java.time stuff. Not that Joda Time inspired built-ins would
> actually require Joda Time, but it is great that many open
> source libraries I use have moved to java.time to eliminate the
> Joda Time dependency, enough that I've been able to
> remove that dependency completely (speaking the Moqui Framework project).
>
> All of this falls into the category of avoiding redundant code and
> standards (interfaces and such). IMO when something in the standard
> Java API is available and adequate it should be preferred over
> alternatives. This lowers the learning curve and has the potential
> to make things run faster and more efficiently as well.

Sure, but FTL isn't a generic JVM programming language, so I don't see
how to apply this on it (see problems above).

> -David

-- 
Thanks,
 Daniel Dekany

Re: [FM3] Package structure changes

[David E Jones <de...@dejc.com>]

On Wed, 2017-01-11 at 11:55 +0100, Daniel Dekany wrote:
> 
> (3) Move TemplateModel interfaces and standard implementations from
> o.a.f to o.a.f.model
> 
> Template models is a topic that's easy to understand for the users
> (mentally and technically separate from the other concerns), so we
> could decrease the surface area of the o.f package API with this.
> 
> As of the TemplateModel implementations, let's say for now that we
> move them over from o.a.f too. Though there will be some more moving
> around in that department, together with BeansWrapper and its friends,
> but that's for another day.

This may also be a topic for another day, but IIRC there was some discussion about getting rid of many of these
interfaces in FM3.

I like the idea of more generally using standard interfaces such as java.util.List and java.util.Map, and even more
generally avoiding the object wrapping that FreeMarker currently does.

This may complicate some of the internal introspection code and possibly even make certain things slower, but overall
avoiding object wrapping helps a lot with memory and garbage collection overhead and makes the template engine more
flexible.

On the topic of using more standard Java interfaces and API: many built-ins can be eliminated in favor of existing API
methods, though perhaps some current FreeMarker users prefer the ?foo syntax of built-ins versus the .foo() syntax of
API calls. For any new built-ins, like the date/time ones recently discussed, they could at least be inspired by the
more standard Java APIs like the java.time classes as opposed to the Joda Time ones.

Joda Time is a great little package, but for stuff I've been doing lately it has been very nice to migrate to the
java.time stuff. Not that Joda Time inspired built-ins would actually require Joda Time, but it is great that many open
source libraries I use have moved to java.time to eliminate the Joda Time dependency, enough that I've been able to
remove that dependency completely (speaking the Moqui Framework project).

All of this falls into the category of avoiding redundant code and standards (interfaces and such). IMO when something
in the standard Java API is available and adequate it should be preferred over alternatives. This lowers the learning
curve and has the potential to make things run faster and more efficiently as well.

-David

Re: [FM3] Package structure changes

[Daniel Dekany <dd...@freemail.hu>]

So, based on the discussion below, I have concluded that the most
basic Maven artifact of FM3 will be "freemarker-core" (so that's
similar to freemarker.jar today, but somewhat leaner, as it doesn't
contain servlet/jsp support for example), and its base package will be
org.apache.freemarker.core. That will enclose some subpackages (like
the equivalent of FM2 freemarker.cache), all of which belongs to
freemarker-core.jar. Basic classes like Configuration, Template,
Environment, will be all directly inside org.apache.freemarker.core.

Any more thoughts on this?


Wednesday, January 11, 2017, 8:58:29 PM, Daniel Dekany wrote:

> Wednesday, January 11, 2017, 3:50:50 PM, Woonsan Ko wrote:
[snip]
>>> (2) Unify freemarker.template and freemarker.core, under o.a.f
>>>
>>> In FM2 we have a separate freemarler.template package, which contains
>>> the most commonly used classes (Configuration, Template, TemplateModel
>>> classes), and there's a freemarker.core package which contains some
>>> less often used ones (such as Environment), and lots of classes that
>>> are public but was marked as internal (AST and such), and of course
>>> non-public internals too. The root "freemarker" package itself is unused.
>>>
>>> There were two recurring problems with this in FM2:
>>>
>>> - What is most- and least commonly used is a blurry line, so it's not
>>>   very helpful classification for the API users. Especially if we hide
>>>   the internal but sadly public classes (and we do that on the JavaDoc
>>>   level even in FM2), the concern (topic) of the two packages isn't
>>>   clear. Both deal with core engine stuff really. (And
>>>   freemarker.template is just a misnomer... at least it never made
>>>   sense to me. Why "template"?)
>>>
>>> - The freemarker.template package and the freemarker.core package needs to
>>>   cross-reference each other a lot (as their concerns aren't well
>>>   separated from engineering point of view either). As Java's
>>>   visibility rules are rather simplistic (and the Java 9 module system
>>>   won't be common place anytime soon), this forces us to make things
>>>   public that shouldn't be that, or to do hacks with "_XxxApi"
>>>   classes.
>>>
>>> Therefor I believe we should just get rid of the freemarker.template
>>> and freemarker.core spearation, and move their contents into o.a.f.
>>
>> Maybe we foresee a possibility to have extension modules in the
>> future. Just as theoretical examples, freemarker-jsp jar,
>> freemarker-spring jar, etc.,
>
> Yes, that's the plan.
>
>> assuming the core can be packaged as freemarker-core jar. For the
>> possibility, wouldn't it help to name the main package name to
>> o.a.freemarker.core instead of o.a.freemarker?
>
> Indeed, maybe that's better, but I'm not sure. Technically both works
> the same (as far as I see). If we just use o.a.freemarker directly,
> because each Maven module can have Java subpackages, it's not obvious
> if o.a.freemarker.foo is just a subpackage inside
> o.a.freemarker:freemarker, or it's inside
> o.a.freemarker:freemarker-foo (unless you happen to know that there's
> a freemarker-foo). So in that sense, o.a.freemarker.core is superior.
> OTOH, I wonder which is the most intuitive for the users.
>
> Any other opinions about this?
[snip]

-- 
Thanks,
 Daniel Dekany

Re: [FM3] Package structure changes

[Daniel Dekany <dd...@freemail.hu>]

Wednesday, January 11, 2017, 8:58:29 PM, Daniel Dekany wrote:

> Wednesday, January 11, 2017, 3:50:50 PM, Woonsan Ko wrote:
[snip]
>> Regarding the internal classes, I'd personally prefer putting into
>> o.a.freemarker.core.internal. It could give a good insight for
>> advanced API users at least.
>
> But that's exactly what I want to avoid. It has caused many problems
> in FM 2. That they are internal means that you can't use them at all,
> because if some people do that, even accidentally (IDE auto
> completion...), we will break their code. Yes their mistake, but
> still... certainly you wont refactor internal code so freely anymore.

I have forgotten to mention here the other difficulty with something
like o.a.f.core.internal. Classes in it will make calls towards
o.a.f.core classes. But as these two package very closely belong
together (they aren't two proper layers on top of each other), some of
those should go to internal methods inside o.a.f.core, but you can't
have public internal methods there for sure (as it's not even an
.internal package).

-- 
Thanks,
 Daniel Dekany

Re: [FM3] Package structure changes

[Daniel Dekany <dd...@freemail.hu>]

Wednesday, January 11, 2017, 3:50:50 PM, Woonsan Ko wrote:

> On Wed, Jan 11, 2017 at 5:55 AM, Daniel Dekany <dd...@freemail.hu> wrote:
>> A bit of prelude first... This is about FreeMarker 3 development (FM3
>> from now on). If life and the community allows it, there will be tons
>> of smaller and quite a few bigger changes in FM3. I intend to drop
>> together a mail like this for most "change sets", so we can see if
>> more discussion (and voting in worse case) is needed. Generally, I
>> will wait for answers for ~72 hours, then even if there was no answer,
>> I will assume that it's accepted (because anything can be undone later
>> after all, it's just extra work).
>>
>> So, to the subject. Please tell your opinion of the points below.
>>
>>
>> (1) Using org.apache.freemarker (o.a.f) as top level package
>
> +1
>
>>
>> As it was discussed long ago, FreeMarker 3 (FM3 from now on) will use
>> a separate parent package from FM2, org.apache.freemarker (o.a.f from
>> now on). (I also had idea that it could be org.apache.freemarker.v3,
>> but org.apache.freemarker is surely look cleaner.) The main point of
>> that is that hence FM2 and FM3 can be used together in the same "class
>> path", which is important because the main point of FM3 is to get rid
>> of the backward compatibility baggage.
>>
>> This was discussed once, but if anyone has anything add, do it.
>>
>>
>> (2) Unify freemarker.template and freemarker.core, under o.a.f
>>
>> In FM2 we have a separate freemarler.template package, which contains
>> the most commonly used classes (Configuration, Template, TemplateModel
>> classes), and there's a freemarker.core package which contains some
>> less often used ones (such as Environment), and lots of classes that
>> are public but was marked as internal (AST and such), and of course
>> non-public internals too. The root "freemarker" package itself is unused.
>>
>> There were two recurring problems with this in FM2:
>>
>> - What is most- and least commonly used is a blurry line, so it's not
>>   very helpful classification for the API users. Especially if we hide
>>   the internal but sadly public classes (and we do that on the JavaDoc
>>   level even in FM2), the concern (topic) of the two packages isn't
>>   clear. Both deal with core engine stuff really. (And
>>   freemarker.template is just a misnomer... at least it never made
>>   sense to me. Why "template"?)
>>
>> - The freemarker.template package and the freemarker.core package needs to
>>   cross-reference each other a lot (as their concerns aren't well
>>   separated from engineering point of view either). As Java's
>>   visibility rules are rather simplistic (and the Java 9 module system
>>   won't be common place anytime soon), this forces us to make things
>>   public that shouldn't be that, or to do hacks with "_XxxApi"
>>   classes.
>>
>> Therefor I believe we should just get rid of the freemarker.template
>> and freemarker.core spearation, and move their contents into o.a.f.
>
> Maybe we foresee a possibility to have extension modules in the
> future. Just as theoretical examples, freemarker-jsp jar,
> freemarker-spring jar, etc.,

Yes, that's the plan.

> assuming the core can be packaged as freemarker-core jar. For the
> possibility, wouldn't it help to name the main package name to
> o.a.freemarker.core instead of o.a.freemarker?

Indeed, maybe that's better, but I'm not sure. Technically both works
the same (as far as I see). If we just use o.a.freemarker directly,
because each Maven module can have Java subpackages, it's not obvious
if o.a.freemarker.foo is just a subpackage inside
o.a.freemarker:freemarker, or it's inside
o.a.freemarker:freemarker-foo (unless you happen to know that there's
a freemarker-foo). So in that sense, o.a.freemarker.core is superior.
OTOH, I wonder which is the most intuitive for the users.

Any other opinions about this?

> (BTW, I have assumed FM3 should structured like a maven multi-module
> project. I guess Gradle should have the same concept.)

Yes.

> Regarding the internal classes, I'd personally prefer putting into
> o.a.freemarker.core.internal. It could give a good insight for
> advanced API users at least.

But that's exactly what I want to avoid. It has caused many problems
in FM 2. That they are internal means that you can't use them at all,
because if some people do that, even accidentally (IDE auto
completion...), we will break their code. Yes their mistake, but
still... certainly you wont refactor internal code so freely anymore.

>> (One the same time we should move out some classes from it, but see
>> later.)
>>
>>
>> (3) Move TemplateModel interfaces and standard implementations from
>> o.a.f to o.a.f.model
>
> +1
> (If o.a.f.core is preferred, that can be o.a.f.core.model.)

(If is o.a.f.core is preferred then yes.)

>> Template models is a topic that's easy to understand for the users
>> (mentally and technically separate from the other concerns), so we
>> could decrease the surface area of the o.f package API with this.
>>
>> As of the TemplateModel implementations, let's say for now that we
>> move them over from o.a.f too. Though there will be some more moving
>> around in that department, together with BeansWrapper and its friends,
>> but that's for another day.
>>
>>
>> (4) Rename freemarker.cache to o.f.templateloader
>
> +1
> (or o.a.f.core.templateloader.)

(Then yes.)

> Regards,
>
> Woonsan
>
>>
>> Because, it was misnomer. Template loading is the main topic of the
>> package (caching of the TemplateLoader-ed templates is part of it).
>>
>> --
>> Thanks,
>>  Daniel Dekany

-- 
Thanks,
 Daniel Dekany

Re: [FM3] Package structure changes

[Woonsan Ko <wo...@apache.org>]

On Wed, Jan 11, 2017 at 5:55 AM, Daniel Dekany <dd...@freemail.hu> wrote:
> A bit of prelude first... This is about FreeMarker 3 development (FM3
> from now on). If life and the community allows it, there will be tons
> of smaller and quite a few bigger changes in FM3. I intend to drop
> together a mail like this for most "change sets", so we can see if
> more discussion (and voting in worse case) is needed. Generally, I
> will wait for answers for ~72 hours, then even if there was no answer,
> I will assume that it's accepted (because anything can be undone later
> after all, it's just extra work).
>
> So, to the subject. Please tell your opinion of the points below.
>
>
> (1) Using org.apache.freemarker (o.a.f) as top level package

+1

>
> As it was discussed long ago, FreeMarker 3 (FM3 from now on) will use
> a separate parent package from FM2, org.apache.freemarker (o.a.f from
> now on). (I also had idea that it could be org.apache.freemarker.v3,
> but org.apache.freemarker is surely look cleaner.) The main point of
> that is that hence FM2 and FM3 can be used together in the same "class
> path", which is important because the main point of FM3 is to get rid
> of the backward compatibility baggage.
>
> This was discussed once, but if anyone has anything add, do it.
>
>
> (2) Unify freemarker.template and freemarker.core, under o.a.f
>
> In FM2 we have a separate freemarler.template package, which contains
> the most commonly used classes (Configuration, Template, TemplateModel
> classes), and there's a freemarker.core package which contains some
> less often used ones (such as Environment), and lots of classes that
> are public but was marked as internal (AST and such), and of course
> non-public internals too. The root "freemarker" package itself is unused.
>
> There were two recurring problems with this in FM2:
>
> - What is most- and least commonly used is a blurry line, so it's not
>   very helpful classification for the API users. Especially if we hide
>   the internal but sadly public classes (and we do that on the JavaDoc
>   level even in FM2), the concern (topic) of the two packages isn't
>   clear. Both deal with core engine stuff really. (And
>   freemarker.template is just a misnomer... at least it never made
>   sense to me. Why "template"?)
>
> - The freemarker.template package and the freemarker.core package needs to
>   cross-reference each other a lot (as their concerns aren't well
>   separated from engineering point of view either). As Java's
>   visibility rules are rather simplistic (and the Java 9 module system
>   won't be common place anytime soon), this forces us to make things
>   public that shouldn't be that, or to do hacks with "_XxxApi"
>   classes.
>
> Therefor I believe we should just get rid of the freemarker.template
> and freemarker.core spearation, and move their contents into o.a.f.

Maybe we foresee a possibility to have extension modules in the
future. Just as theoretical examples, freemarker-jsp jar,
freemarker-spring jar, etc., assuming the core can be packaged as
freemarker-core jar.
For the possibility, wouldn't it help to name the main package name to
o.a.freemarker.core instead of o.a.freemarker?
(BTW, I have assumed FM3 should structured like a maven multi-module
project. I guess Gradle should have the same concept.)

Regarding the internal classes, I'd personally prefer putting into
o.a.freemarker.core.internal. It could give a good insight for
advanced API users at least.

> (One the same time we should move out some classes from it, but see
> later.)
>
>
> (3) Move TemplateModel interfaces and standard implementations from
> o.a.f to o.a.f.model

+1
(If o.a.f.core is preferred, that can be o.a.f.core.model.)

>
> Template models is a topic that's easy to understand for the users
> (mentally and technically separate from the other concerns), so we
> could decrease the surface area of the o.f package API with this.
>
> As of the TemplateModel implementations, let's say for now that we
> move them over from o.a.f too. Though there will be some more moving
> around in that department, together with BeansWrapper and its friends,
> but that's for another day.
>
>
> (4) Rename freemarker.cache to o.f.templateloader

+1
(or o.a.f.core.templateloader.)

Regards,

Woonsan

>
> Because, it was misnomer. Template loading is the main topic of the
> package (caching of the TemplateLoader-ed templates is part of it).
>
> --
> Thanks,
>  Daniel Dekany
>