The problem of "too human" text transformations

[Daniel Dekany <dd...@apache.org>]

There's this old frequently requested built-in,
maybeTooLongText?truncate(n). Like "No space for such a long
title"?truncate(13) would give "No space...". The problem with these
is that there's no single algorithm for this that fits the need of
everyone. Like maybe the result should be "No space f...", or "No
space [...]", etc. This can even change depending on the current value
of the `locale` FreeMarker setting.

Now there's another similar wish, for "?title_case", which has the
same problem, plus it's much more locale dependent. See:

https://issues.apache.org/jira/browse/FREEMARKER-97

We do provide locale dependent functions, like number and date
formatting. This is pretty much an FM tradition, to treat non-English
users as equals (well, except that the keywords aren't localizable).
However, we push those tasks to the Java platform. But it doesn't
support title case (as far as I know). Besides there can be different
preferences even inside the same country, just as with ?truncate.

So, it's acceptable to chicken out ;) saying that none of the above
has a widely accepted default, so the correct solution is letting the
user solve this. But I think in practice what happens is that people
need the functionality (especially ?truncate), and then has to come up
with some ad-hoc solution on the spot, which will be almost always
worse than the default we could provide. So we might as well just
provide some decent default, sparing time for lot of users. Then, then
if they are particular about the algorithm (which is certainly rare
nowadays), let them plug their own via the Configurable API. So there
would be something like
cfg.setTruncateImplementation(localToImplementationMap), where the
parameter maps locale-s to Java functions and defines a fallback
function. (I haven't worked out the API details, as you can see.)

What do you think?

-- 
Thanks,
 Daniel Dekany

Re: The problem of "too human" text transformations

[Daniel Dekany <dd...@apache.org>]

Back to topic... let's not suck here. In FreeMarker 2.x we already
have a lot of built-ins, so I think it doesn't make sense to miss a
few more that's actually asked for, only because for those we should
also support overriding the default implementations, as it's harder to
find a default that satisfies the 99% of users. Maybe FM3 should do
this differently, but the FM2 way of doing this is already what it is
(i.e., users need to do "x"? We add `?x`... problem solved.)


Thursday, May 31, 2018, 11:03:09 AM, Daniel Dekany wrote:

> Thursday, May 31, 2018, 1:16:26 AM, David E Jones wrote:
>
>> My general opinion on this sort of thing is there are lots of libraries and
>> tools around in the Java world for doing string manipulations and such,
>> they don't need to be built into a template language. Taking that one step
>> further, I'd actually prefer that the expression syntax in the template
>> language makes it easy to call an existing API vs trying to make everything
>> built into the template language.
>>
>> For example in addition to using FTL I also use Groovy heavily and Groovy
>> already has a great API including standard Java API augmentation for all
>> sorts of things. Yes, FTL could use the Groovy String and other expanded
>> objects as a reference for built in design and have a good set of commonly
>> needed things, but why not just defer to Groovy for those that use it or
>> whatever other API/library anyone might prefer.
>>
>> I know this is a VERY different direction from FM2. What I find most useful
>> in FTL is features like macros and the extensive directives and other
>> things that make a template language flexible and powerful. The built-ins
>> are handy sometimes, but often don't do what I need anyway so other than
>> things like ?has_content I don't use them a lot.
>
> An important (and as far as I'm concerned, the main) reason for many
> built-ins is/was that frameworks and other products drop in FreeMarker
> for templates and then often don't give much attention to give tools
> to the template authors. Then the poor people who actually has to
> create templates come and ask, "I must do X... how can I do that in
> FreeMaker?". The less often we have to say "Please pester the
> developers (who are often from another company, or are on another
> sub-project by now => you have near 0 chance) to expose such
> functionality to the templates", the better. Instead we can say "Like
> foo?x", and furthermore, ?x is documented, and people can actually
> find its documentation, even years later. Surely if you have some
> small company with the sharp people, this is not a problem for you.
> Actually, then you very often don't want the one-size-fits-all
> solutions anyway.
>
>> I've mentioned this before but what would be a killer feature for FTL is to
>> support pluggable expression evaluators. The FTL expression syntax isn't
>> awful
>
> (Yes it is... but that's besides the point until I have fixed it in
> FM3.)
>
>> but is not nearly as convenient as Groovy, especially when you are
>> using it extensively elsewhere.
>
> The theoretical problem is of course that both languages (the "outer"
> template language and the language from where you rip off the
> expression language from) have its own ideas, and so there will be
> mismatches. The expression language in a language and other parts of
> the language aren't separate in general anyway. Which leads to the
> question that if you are using Groovy expressions, why not also use
> for loops and whatnot from Groovy? (This is fairly common way of
> creating template language, just taking an existing language, and add
> some convenience syntax to embed it into the static text.) So then we
> don't need our own #list, #if, etc. implementation either. But then,
> the original idea was that a template language is a domain specific
> language, and as such can have more specialized, say, loop constructs
> for that domain. (I'm not saying that FM2 did great there, but there
> was a lack of determination there as well.) If that's not the case
> though...
>
> Anyway, when and if FM3 will be more mature, we can see if how doable
> this is in reality. It's also a question if we are talking about a
> syntax, that in 99% looks like Groovy, or we are talking about actual
> Groovy. In the last case TemplateModel-s and the particular way null
> handling will work will be an obstacle for sure. Not to mention
> TemplateMarkupOutputModel-s, that AFAIK don't even have commonly used
> equivalent (like a class in the standard Java API).
>
>> If the default FTL expressions were not used you couldn't use
>> built-ins and instead would just have expressions that are expected
>> to evaluate to a boolean, string, or arbitrary object and can be
>> whatever.
>
> At least in FM3 built-ins will be just functions you have defined in
> the "core" FTL namespace (whose content is specified in template
> language Dialect you use) that's "statically linked" to the templates
> (allowing some parse-time checks and somewhat faster calls). So
> `foo?bar(1)` is basically just a shorthand syntax for `<#import core =
> 'freemarker://dialect-or-something'>` and `core.bar(foo, 1)`. There's
> a standard/default Dialect of course, which has the big advantage that
> it's documented in the official Manual, is used in SO answers, etc.
>
>> How important is this? Important enough that I've
>> considered using a different template language. If FTL wasn't so
>> darn convenience for XML transformation using macros I'd probably be
>> using Groovy GString templates instead, and just for the far better
>> syntax in Groovy.
>>
>> This also goes back to the object wrapping discussion which is another can
>> of worms...
>
> Yeah, it's one of the things that I really wanted to get rid of, but
> it's a very disrupting transition to do. I don't mean for most of the
> users, but for the source code. On the brighter side, what really
> happens when you do that transition is that instead of something like:
>
>   if (value is TemplateMapLikeModel) {
>       ... ((TemplateMapLikeModel) value).get(k) ...
>   }
>
> you, in first approach, rewrite it to something like:
>
>   if (mop.isMapLike(value)) {
>       ... mop.get(value, k) ...
>   }
>
> Of course that would be a silly MOP API design, so this below is more
> likely, but now it's visible that one comes from the other trivially:
>
>
>   MapLike m = mop.tryGetAsMapLike(value);
>   if (m != null) {
>       ... m.get(k) ...
>   }
>
> Hence when we have our TemplateModel-s cleaned up, it's easier to
> migrate to the MOP approach. The "type system" (just a mental model
> really) depicted be the TemplateModel still exists, you just use a MOP
> to implement it.
>
>> I don't mean to derail this discussion, but especially these days in a
>> world of a huge number of great Java libraries out there it is often better
>> to not try to do everything and instead focus on core strengths that you
>> can't get anywhere else, and make it easy for people to use other open
>> source projects for their core strengths. This could even go as far as
>> FreeMarker builds that have dependencies on certain commonly used open
>> source projects, there are many even in the ASF that do various things that
>> FTL built-ins do but with far more options and well known behavior for the
>> many who use them (like various Commons projects).
>
> It's really the duty of the Dialects feature of FM3. You can put
> together your "toolbox" from whatever you want, like call Apache
> Commons. I mean it's unrelated to question of expression languages, as
> far as I see. (Like, you can call plain Java methods from FM2 as
> well.)
>
> If we want to use Apache Commons etc. in implementing the functions of
> the standard dialect... I *guess* we won't benefit from it much at
> this point, but I need to see it case-by-case to tell, like, the FM
> implementation of function x sucks, while Apache Commons StringUtils.x
> is great, etc. In any case, users don't care, as far as they can call
> ?x and it does its job. My point in this thread is that we should be
> able to tell users to "just write ?x" (see earlier), instead of add
> your own solution somehow. (Also, if ?x works, they prefer if we have
> less dependencies. Especially if we want to support Andorid.)
>
>> Would supporting different expression languages cause huge problems for IDE
>> support? Possibly, but IDEs these days have lots of support for language
>> injection and layers that are meant for just this sort of thing.
>
> At least looking from big enough distance it's easy. :) Note that so
> far we fail to deliver an editor for Eclipse that integrates the
> existing HTML editor functionality with FTL... :-> Please everyone go
> and help out Angelo Zerr so that this will change:
>
> https://github.com/angelozerr/lsp4e-freemarker
> https://github.com/angelozerr/freemarker-languageserver
>
> Anyway, IDE support won't be the obstacle. At worse only the default
> expression syntax will have reasonable IDE support.
>
>> In theory FreeMarker itself could be a heck of a lot smaller and
>> simpler and at the same time be far more flexible and powerful. IMO
>> that could be a game changer in the world of Java-based template
>> languages. Some of this would be more work than not supporting this
>> sort of flexibility (ie support Groovy syntax as an option, ie
>> without requiring the much larger Groovy jar files as dependencies
>> by having a default built-in expression syntax), but by simplifying
>> other things it would save a heck of a lot of work (including design
>> work).
>
> So, I'm not sure I follow what exactly are you proposing. To use
> vanilla Groovy (the actual implementation) for expressions? How does
> that blend with the directives (which mostly also have similar pairs
> in Groovy), and other (planned) FM features? (And... why's the result
> called FreeMarker at all? ;) )
>
>> -David
>>
>>
>>
>> On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <slidingfilaments@gmail.com
>>> wrote:
>>
>>> I'm not a subject matter expert here, but I always thought sensible
>>> defaults with the ability to override using configuration is a wise
>>> move because you appeal to many people out of the box and then allow
>>> minority to also not feel locked out when they need to override. Also,
>>> as a general rule, I found from the projects we worked on that
>>> internationalization is always a challenge, but you can also resolve
>>> that with the same solution (sensible defaults with configuration
>>> overrides); then you simply let the community adopt and improve the
>>> code for non-English parts.
>>>
>>> Of course then the challenge becomes, what's a sensible default? I
>>> wish I had a dollar every time I asked that! I'd favor simplicity
>>> where possible. In your example above maybe "No space f..." is the
>>> simplest, and then people can get fancy if they want to?
>>>
>>> My 2 cents, sorry for the noise :)
>>>
>>> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <dd...@apache.org>
>>> wrote:
>>> > There's this old frequently requested built-in,
>>> > maybeTooLongText?truncate(n). Like "No space for such a long
>>> > title"?truncate(13) would give "No space...". The problem with these
>>> > is that there's no single algorithm for this that fits the need of
>>> > everyone. Like maybe the result should be "No space f...", or "No
>>> > space [...]", etc. This can even change depending on the current value
>>> > of the `locale` FreeMarker setting.
>>> >
>>> > Now there's another similar wish, for "?title_case", which has the
>>> > same problem, plus it's much more locale dependent. See:
>>> >
>>> > https://issues.apache.org/jira/browse/FREEMARKER-97
>>> >
>>> > We do provide locale dependent functions, like number and date
>>> > formatting. This is pretty much an FM tradition, to treat non-English
>>> > users as equals (well, except that the keywords aren't localizable).
>>> > However, we push those tasks to the Java platform. But it doesn't
>>> > support title case (as far as I know). Besides there can be different
>>> > preferences even inside the same country, just as with ?truncate.
>>> >
>>> > So, it's acceptable to chicken out ;) saying that none of the above
>>> > has a widely accepted default, so the correct solution is letting the
>>> > user solve this. But I think in practice what happens is that people
>>> > need the functionality (especially ?truncate), and then has to come up
>>> > with some ad-hoc solution on the spot, which will be almost always
>>> > worse than the default we could provide. So we might as well just
>>> > provide some decent default, sparing time for lot of users. Then, then
>>> > if they are particular about the algorithm (which is certainly rare
>>> > nowadays), let them plug their own via the Configurable API. So there
>>> > would be something like
>>> > cfg.setTruncateImplementation(localToImplementationMap), where the
>>> > parameter maps locale-s to Java functions and defines a fallback
>>> > function. (I haven't worked out the API details, as you can see.)
>>> >
>>> > What do you think?
>>> >
>>> > --
>>> > Thanks,
>>> >  Daniel Dekany
>>> >
>>>
>

-- 
Thanks,
 Daniel Dekany

Re: The problem of "too human" text transformations

[Taher Alkhateeb <sl...@gmail.com>]

I haven't done any extensive testing on the built-in constructs for Arabic
support but from my work in OFBiz I can safely say that most of my needs
are already available OOTB. I think this is in part due to the java
platform as well as using the timezone and locale APIs heavily and all
throughout.

On Fri, Jun 8, 2018, 11:48 PM Daniel Dekany <dd...@apache.org> wrote:

> Friday, June 8, 2018, 8:41:34 PM, Taher Alkhateeb wrote:
>
> > One of the absolute hardest things for me to accomplish is to get the
> > Arabic language with all the RTL quirks done right. I'm an Arab but even
> I
> > suffer from the complexity of this language.
>
> Speaking of which, I never hear about the needs of languages that are
> very different (I guess) from the typical western European and US
> languages, such as the Arabic, Chinese, Hindi and languages. I only
> speak Hungarian and English (and has some very dim past experience
> with Russian and German), so I have no idea if FM can be of any help
> there.
>
> > Hoping to achieve true and full internationalization is something that
> > might not be very realistic IMHO. So maybe the best decision is not to
> > support all languages but rather having the architecture and design of
> your
> > templating engine done in an extinsible-friendly manner, which seems to
> be
> > the case here?
>
> Exactly.
>
> I wonder how well the existing features work there, where we build on
> the the localization features of the Java platform. I'm not sure how
> well it works for languages that are very far from the "western
> culture". (I had some feedback that suggests that for some such
> country the defaults are wrong, but for a while it's possible to
> handle that with custom number and date/time formats.)
>
> > So yeah, perhaps I would prefer to go for English OOTB support and keep a
> > flexible architecture in place.
> >
> > On Thu, Jun 7, 2018, 5:34 PM Daniel Dekany <dd...@apache.org> wrote:
> >
> >> Tuesday, June 5, 2018, 3:28:45 PM, Woonsan Ko wrote:
> >>
> >> > On Mon, Jun 4, 2018 at 5:04 AM, Daniel Dekany <dd...@apache.org>
> >> wrote:
> >> >> Sunday, June 3, 2018, 10:13:04 PM, David E Jones wrote:
> >> >>
> >> >>> You make good points Daniel. At a high level it reminds me of a
> >> discussion
> >> >>> with a new client a couple months ago on the topic of managing
> >> templates
> >> >>> for ecommerce sites. They want to use the Liquid template engine
> partly
> >> >>> because they are doing custom ecommerce for some customers and
> Shopify
> >> >>> (which uses Liquid, where Liquid came from) for others, and also
> >> because
> >> >>> Liquid is more constrained and easier for web developers/designers
> to
> >> use.
> >> >>>
> >> >>> There is a case for template languages that are less developer
> >> oriented and
> >> >>> more designer oriented, or maybe better put because visual web
> >> designers
> >> >>> have no business mucking around with templates server side
> developer vs
> >> >>> web/browser developer (even if this is stuff that generally runs on
> a
> >> >>> server anyway).
> >> >>>
> >> >>> Right now, ie FM2, FreeMarker strikes a good balance between these
> (IMO
> >> >>> anyway), and the built-ins are an important part of that. Maybe they
> >> don't
> >> >>> do every little thing everyone in the world might want but they
> cover
> >> a lot
> >> >>> of basics and common needs.
> >> >>>
> >> >>> At a lower level you are correct about template parsing and
> supporting
> >> >>> arbitrary languages for inline expressions. The template parser
> still
> >> needs
> >> >>> to be able to figure out where the expression starts and stops. This
> >> might
> >> >>> not be too bad for C-like languages which current FM expressions are
> >> (more
> >> >>> or less, mostly the same demarcation characters and all), but is
> still
> >> a
> >> >>> challenge and could require something like surrounding expressions
> with
> >> >>> parenthesis (along with nested parenthesis evaluation that FM
> already
> >> does)
> >> >>> and that's a little bit ugly.
> >> >>
> >> >> I wasn't even worried about that yet, but the semantical differences.
> >> >> The expression and non-expression language (if they are separate at
> >> >> all) build on a common mental model, so you can't, in general, cut
> out
> >> >> the expression language from one language, and put it into another.
> So
> >> >> I can only look at this on a case-by-case basis.
> >> >>
> >> >> As of the syntactical problems, at least if I write the expression
> >> >> parser, I can just exit expression mode if the next token would lead
> >> >> to an error, and let the "outer language" process the next character.
> >> >> Like you have `a + b n=2`, then `n` would be an error, and the outer
> >> >> parser will pick it up and recognize that as a parameter assignment.
> >> >> What's tricky is making JavaCC to do this... it doesn't support
> >> >> dynamic composing by design. ANTLR, I don't know (nor if it's fast
> >> >> enough for use cases like a lot of ?eval/?interpret). For a hand
> >> >> written parser, it's certainly not a problem. Though switching to
> hand
> >> >> written is a questionable step, even if I personally find it
> tempting.
> >> >>
> >> >>> Anyway, sorry to derail the discussion.
> >> >>>
> >> >>> On the truncate built-in an optional String type second argument
> makes
> >> >>> sense as a suffix to use if the string is truncated, ie that seems
> >> like an
> >> >>> easy thing to accommodate without reducing flexibility or having to
> >> write
> >> >>> that logic to do that inline in the template.
> >> >>
> >> >> There's also the question of where to cut the text. There's the
> >> >> maximum length given as parameter (actually, you subtract the length
> >> >> of the cut marker suffix), however if with applying that naively you
> >> >> end up with white-space at the end, you probably want to trim that.
> >> >> Also certainly if it ends with a ellipsis, dot or a sequence of dots,
> >> >> and maybe even if with some other punctuation. There's also the
> >> >> typical preference, over a certain maximum length, where you only cut
> >> >> at the end of words (unless you have some extremely long word).
> >> >
> >> > Commons-text's WordUtils#abbreviate(str, lower, upper, appendToEnd)
> >> > finds a space from the lower index to find where to cut.
> >> > It doesn't care other white spaces nor ending dot(s), but one of
> >> > existing simple algorithms at the moment. ;-)
> >> >
> >> >>
> >> >>> On title casing it might be best to defer to what Java does (ie
> >> >>> Character.toUpperCase() or .toTitleCase()), or were there issues
> with
> >> that?
> >> >>
> >> >> They don't convert text to title case, only a single character. In
> the
> >> >> most basic form of title case conversion of text, you convert the
> >> >> first letter of each word to title case. You may already have some
> >> >> tricky parts there with finding word boundaries. Then, in many
> >> >> languages, like in English, you aren't supported to convert words
> like
> >> >> "and", "of", etc. So it's obviously locale dependent.
> >> >
> >> > WordUtils#capitalizeFully() does. It doesn't recognize "and", "of",
> etc.
> >> though.
> >>
> >> Both WordUtils methods seem to be too simplistic for real world usage,
> >> to me at least. The default implementation should be actually usable
> >> for at least English content, and beat typical quick in-house
> >> solutions in quality. It seems that that's doable (see Jira issue).
> >>
> >> I guess the other languages will be left for the users (the algorithm
> >> is pluggable per language). As far as I know the most common language
> >> used for web content is still English, by far. So I guess it's the
> >> language FM is used for the most as well. It's quite weird as English
> >> isn't the most widely spoken language among the people who use the
> >> Internet. But I'm afraid if we pick, let's say, the next 4 most common
> >> web content languages, that can be be more offending for those left
> >> out, than if we support English only, which people more or less have
> >> get used to.
> >>
> >> > But as you said, if it can be customized through Configuration for
> >> > locale specific use cases, I'd say it's perfect.
> >> >
> >> > Regards,
> >> >
> >> > Woonsan
> >> >
> >> >>
> >> >>> -David
> >> >>>
> >> >>>
> >> >>> On Thu, May 31, 2018 at 2:03 AM, Daniel Dekany <dd...@apache.org>
> >> wrote:
> >> >>>
> >> >>>> Thursday, May 31, 2018, 1:16:26 AM, David E Jones wrote:
> >> >>>>
> >> >>>> > My general opinion on this sort of thing is there are lots of
> >> libraries
> >> >>>> and
> >> >>>> > tools around in the Java world for doing string manipulations and
> >> such,
> >> >>>> > they don't need to be built into a template language. Taking that
> >> one
> >> >>>> step
> >> >>>> > further, I'd actually prefer that the expression syntax in the
> >> template
> >> >>>> > language makes it easy to call an existing API vs trying to make
> >> >>>> everything
> >> >>>> > built into the template language.
> >> >>>> >
> >> >>>> > For example in addition to using FTL I also use Groovy heavily
> and
> >> Groovy
> >> >>>> > already has a great API including standard Java API augmentation
> >> for all
> >> >>>> > sorts of things. Yes, FTL could use the Groovy String and other
> >> expanded
> >> >>>> > objects as a reference for built in design and have a good set of
> >> >>>> commonly
> >> >>>> > needed things, but why not just defer to Groovy for those that
> use
> >> it or
> >> >>>> > whatever other API/library anyone might prefer.
> >> >>>> >
> >> >>>> > I know this is a VERY different direction from FM2. What I find
> most
> >> >>>> useful
> >> >>>> > in FTL is features like macros and the extensive directives and
> >> other
> >> >>>> > things that make a template language flexible and powerful. The
> >> built-ins
> >> >>>> > are handy sometimes, but often don't do what I need anyway so
> other
> >> than
> >> >>>> > things like ?has_content I don't use them a lot.
> >> >>>>
> >> >>>> An important (and as far as I'm concerned, the main) reason for
> many
> >> >>>> built-ins is/was that frameworks and other products drop in
> FreeMarker
> >> >>>> for templates and then often don't give much attention to give
> tools
> >> >>>> to the template authors. Then the poor people who actually has to
> >> >>>> create templates come and ask, "I must do X... how can I do that in
> >> >>>> FreeMaker?". The less often we have to say "Please pester the
> >> >>>> developers (who are often from another company, or are on another
> >> >>>> sub-project by now => you have near 0 chance) to expose such
> >> >>>> functionality to the templates", the better. Instead we can say
> "Like
> >> >>>> foo?x", and furthermore, ?x is documented, and people can actually
> >> >>>> find its documentation, even years later. Surely if you have some
> >> >>>> small company with the sharp people, this is not a problem for you.
> >> >>>> Actually, then you very often don't want the one-size-fits-all
> >> >>>> solutions anyway.
> >> >>>>
> >> >>>> > I've mentioned this before but what would be a killer feature for
> >> FTL is
> >> >>>> to
> >> >>>> > support pluggable expression evaluators. The FTL expression
> syntax
> >> isn't
> >> >>>> > awful
> >> >>>>
> >> >>>> (Yes it is... but that's besides the point until I have fixed it in
> >> >>>> FM3.)
> >> >>>>
> >> >>>> > but is not nearly as convenient as Groovy, especially when you
> are
> >> >>>> > using it extensively elsewhere.
> >> >>>>
> >> >>>> The theoretical problem is of course that both languages (the
> "outer"
> >> >>>> template language and the language from where you rip off the
> >> >>>> expression language from) have its own ideas, and so there will be
> >> >>>> mismatches. The expression language in a language and other parts
> of
> >> >>>> the language aren't separate in general anyway. Which leads to the
> >> >>>> question that if you are using Groovy expressions, why not also use
> >> >>>> for loops and whatnot from Groovy? (This is fairly common way of
> >> >>>> creating template language, just taking an existing language, and
> add
> >> >>>> some convenience syntax to embed it into the static text.) So then
> we
> >> >>>> don't need our own #list, #if, etc. implementation either. But
> then,
> >> >>>> the original idea was that a template language is a domain specific
> >> >>>> language, and as such can have more specialized, say, loop
> constructs
> >> >>>> for that domain. (I'm not saying that FM2 did great there, but
> there
> >> >>>> was a lack of determination there as well.) If that's not the case
> >> >>>> though...
> >> >>>>
> >> >>>> Anyway, when and if FM3 will be more mature, we can see if how
> doable
> >> >>>> this is in reality. It's also a question if we are talking about a
> >> >>>> syntax, that in 99% looks like Groovy, or we are talking about
> actual
> >> >>>> Groovy. In the last case TemplateModel-s and the particular way
> null
> >> >>>> handling will work will be an obstacle for sure. Not to mention
> >> >>>> TemplateMarkupOutputModel-s, that AFAIK don't even have commonly
> used
> >> >>>> equivalent (like a class in the standard Java API).
> >> >>>>
> >> >>>> > If the default FTL expressions were not used you couldn't use
> >> >>>> > built-ins and instead would just have expressions that are
> expected
> >> >>>> > to evaluate to a boolean, string, or arbitrary object and can be
> >> >>>> > whatever.
> >> >>>>
> >> >>>> At least in FM3 built-ins will be just functions you have defined
> in
> >> >>>> the "core" FTL namespace (whose content is specified in template
> >> >>>> language Dialect you use) that's "statically linked" to the
> templates
> >> >>>> (allowing some parse-time checks and somewhat faster calls). So
> >> >>>> `foo?bar(1)` is basically just a shorthand syntax for `<#import
> core =
> >> >>>> 'freemarker://dialect-or-something'>` and `core.bar(foo, 1)`.
> There's
> >> >>>> a standard/default Dialect of course, which has the big advantage
> that
> >> >>>> it's documented in the official Manual, is used in SO answers, etc.
> >> >>>>
> >> >>>> > How important is this? Important enough that I've
> >> >>>> > considered using a different template language. If FTL wasn't so
> >> >>>> > darn convenience for XML transformation using macros I'd
> probably be
> >> >>>> > using Groovy GString templates instead, and just for the far
> better
> >> >>>> > syntax in Groovy.
> >> >>>> >
> >> >>>> > This also goes back to the object wrapping discussion which is
> >> another
> >> >>>> can
> >> >>>> > of worms...
> >> >>>>
> >> >>>> Yeah, it's one of the things that I really wanted to get rid of,
> but
> >> >>>> it's a very disrupting transition to do. I don't mean for most of
> the
> >> >>>> users, but for the source code. On the brighter side, what really
> >> >>>> happens when you do that transition is that instead of something
> like:
> >> >>>>
> >> >>>>   if (value is TemplateMapLikeModel) {
> >> >>>>       ... ((TemplateMapLikeModel) value).get(k) ...
> >> >>>>   }
> >> >>>>
> >> >>>> you, in first approach, rewrite it to something like:
> >> >>>>
> >> >>>>   if (mop.isMapLike(value)) {
> >> >>>>       ... mop.get(value, k) ...
> >> >>>>   }
> >> >>>>
> >> >>>> Of course that would be a silly MOP API design, so this below is
> more
> >> >>>> likely, but now it's visible that one comes from the other
> trivially:
> >> >>>>
> >> >>>>
> >> >>>>   MapLike m = mop.tryGetAsMapLike(value);
> >> >>>>   if (m != null) {
> >> >>>>       ... m.get(k) ...
> >> >>>>   }
> >> >>>>
> >> >>>> Hence when we have our TemplateModel-s cleaned up, it's easier to
> >> >>>> migrate to the MOP approach. The "type system" (just a mental model
> >> >>>> really) depicted be the TemplateModel still exists, you just use a
> MOP
> >> >>>> to implement it.
> >> >>>>
> >> >>>> > I don't mean to derail this discussion, but especially these days
> >> in a
> >> >>>> > world of a huge number of great Java libraries out there it is
> often
> >> >>>> better
> >> >>>> > to not try to do everything and instead focus on core strengths
> >> that you
> >> >>>> > can't get anywhere else, and make it easy for people to use other
> >> open
> >> >>>> > source projects for their core strengths. This could even go as
> far
> >> as
> >> >>>> > FreeMarker builds that have dependencies on certain commonly used
> >> open
> >> >>>> > source projects, there are many even in the ASF that do various
> >> things
> >> >>>> that
> >> >>>> > FTL built-ins do but with far more options and well known
> behavior
> >> for
> >> >>>> the
> >> >>>> > many who use them (like various Commons projects).
> >> >>>>
> >> >>>> It's really the duty of the Dialects feature of FM3. You can put
> >> >>>> together your "toolbox" from whatever you want, like call Apache
> >> >>>> Commons. I mean it's unrelated to question of expression
> languages, as
> >> >>>> far as I see. (Like, you can call plain Java methods from FM2 as
> >> >>>> well.)
> >> >>>>
> >> >>>> If we want to use Apache Commons etc. in implementing the
> functions of
> >> >>>> the standard dialect... I *guess* we won't benefit from it much at
> >> >>>> this point, but I need to see it case-by-case to tell, like, the FM
> >> >>>> implementation of function x sucks, while Apache Commons
> StringUtils.x
> >> >>>> is great, etc. In any case, users don't care, as far as they can
> call
> >> >>>> ?x and it does its job. My point in this thread is that we should
> be
> >> >>>> able to tell users to "just write ?x" (see earlier), instead of add
> >> >>>> your own solution somehow. (Also, if ?x works, they prefer if we
> have
> >> >>>> less dependencies. Especially if we want to support Andorid.)
> >> >>>>
> >> >>>> > Would supporting different expression languages cause huge
> problems
> >> for
> >> >>>> IDE
> >> >>>> > support? Possibly, but IDEs these days have lots of support for
> >> language
> >> >>>> > injection and layers that are meant for just this sort of thing.
> >> >>>>
> >> >>>> At least looking from big enough distance it's easy. :) Note that
> so
> >> >>>> far we fail to deliver an editor for Eclipse that integrates the
> >> >>>> existing HTML editor functionality with FTL... :-> Please everyone
> go
> >> >>>> and help out Angelo Zerr so that this will change:
> >> >>>>
> >> >>>> https://github.com/angelozerr/lsp4e-freemarker
> >> >>>> https://github.com/angelozerr/freemarker-languageserver
> >> >>>>
> >> >>>> Anyway, IDE support won't be the obstacle. At worse only the
> default
> >> >>>> expression syntax will have reasonable IDE support.
> >> >>>>
> >> >>>> > In theory FreeMarker itself could be a heck of a lot smaller and
> >> >>>> > simpler and at the same time be far more flexible and powerful.
> IMO
> >> >>>> > that could be a game changer in the world of Java-based template
> >> >>>> > languages. Some of this would be more work than not supporting
> this
> >> >>>> > sort of flexibility (ie support Groovy syntax as an option, ie
> >> >>>> > without requiring the much larger Groovy jar files as
> dependencies
> >> >>>> > by having a default built-in expression syntax), but by
> simplifying
> >> >>>> > other things it would save a heck of a lot of work (including
> design
> >> >>>> > work).
> >> >>>>
> >> >>>> So, I'm not sure I follow what exactly are you proposing. To use
> >> >>>> vanilla Groovy (the actual implementation) for expressions? How
> does
> >> >>>> that blend with the directives (which mostly also have similar
> pairs
> >> >>>> in Groovy), and other (planned) FM features? (And... why's the
> result
> >> >>>> called FreeMarker at all? ;) )
> >> >>>>
> >> >>>> > -David
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <
> >> >>>> slidingfilaments@gmail.com
> >> >>>> >> wrote:
> >> >>>> >
> >> >>>> >> I'm not a subject matter expert here, but I always thought
> sensible
> >> >>>> >> defaults with the ability to override using configuration is a
> wise
> >> >>>> >> move because you appeal to many people out of the box and then
> >> allow
> >> >>>> >> minority to also not feel locked out when they need to override.
> >> Also,
> >> >>>> >> as a general rule, I found from the projects we worked on that
> >> >>>> >> internationalization is always a challenge, but you can also
> >> resolve
> >> >>>> >> that with the same solution (sensible defaults with
> configuration
> >> >>>> >> overrides); then you simply let the community adopt and improve
> the
> >> >>>> >> code for non-English parts.
> >> >>>> >>
> >> >>>> >> Of course then the challenge becomes, what's a sensible
> default? I
> >> >>>> >> wish I had a dollar every time I asked that! I'd favor
> simplicity
> >> >>>> >> where possible. In your example above maybe "No space f..." is
> the
> >> >>>> >> simplest, and then people can get fancy if they want to?
> >> >>>> >>
> >> >>>> >> My 2 cents, sorry for the noise :)
> >> >>>> >>
> >> >>>> >> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <
> >> ddekany@apache.org>
> >> >>>> >> wrote:
> >> >>>> >> > There's this old frequently requested built-in,
> >> >>>> >> > maybeTooLongText?truncate(n). Like "No space for such a long
> >> >>>> >> > title"?truncate(13) would give "No space...". The problem with
> >> these
> >> >>>> >> > is that there's no single algorithm for this that fits the
> need
> >> of
> >> >>>> >> > everyone. Like maybe the result should be "No space f...", or
> "No
> >> >>>> >> > space [...]", etc. This can even change depending on the
> current
> >> value
> >> >>>> >> > of the `locale` FreeMarker setting.
> >> >>>> >> >
> >> >>>> >> > Now there's another similar wish, for "?title_case", which has
> >> the
> >> >>>> >> > same problem, plus it's much more locale dependent. See:
> >> >>>> >> >
> >> >>>> >> > https://issues.apache.org/jira/browse/FREEMARKER-97
> >> >>>> >> >
> >> >>>> >> > We do provide locale dependent functions, like number and date
> >> >>>> >> > formatting. This is pretty much an FM tradition, to treat
> >> non-English
> >> >>>> >> > users as equals (well, except that the keywords aren't
> >> localizable).
> >> >>>> >> > However, we push those tasks to the Java platform. But it
> doesn't
> >> >>>> >> > support title case (as far as I know). Besides there can be
> >> different
> >> >>>> >> > preferences even inside the same country, just as with
> ?truncate.
> >> >>>> >> >
> >> >>>> >> > So, it's acceptable to chicken out ;) saying that none of the
> >> above
> >> >>>> >> > has a widely accepted default, so the correct solution is
> >> letting the
> >> >>>> >> > user solve this. But I think in practice what happens is that
> >> people
> >> >>>> >> > need the functionality (especially ?truncate), and then has to
> >> come up
> >> >>>> >> > with some ad-hoc solution on the spot, which will be almost
> >> always
> >> >>>> >> > worse than the default we could provide. So we might as well
> just
> >> >>>> >> > provide some decent default, sparing time for lot of users.
> >> Then, then
> >> >>>> >> > if they are particular about the algorithm (which is certainly
> >> rare
> >> >>>> >> > nowadays), let them plug their own via the Configurable API.
> So
> >> there
> >> >>>> >> > would be something like
> >> >>>> >> > cfg.setTruncateImplementation(localToImplementationMap), where
> >> the
> >> >>>> >> > parameter maps locale-s to Java functions and defines a
> fallback
> >> >>>> >> > function. (I haven't worked out the API details, as you can
> see.)
> >> >>>> >> >
> >> >>>> >> > What do you think?
> >> >>>> >> >
> >> >>>> >> > --
> >> >>>> >> > Thanks,
> >> >>>> >> >  Daniel Dekany
> >> >>>> >> >
> >> >>>> >>
> >> >>>>
> >> >>>> --
> >> >>>> Thanks,
> >> >>>>  Daniel Dekany
> >> >>>>
> >> >>>>
> >> >>
> >> >> --
> >> >> Thanks,
> >> >>  Daniel Dekany
> >> >>
> >> >
> >>
> >> --
> >> Thanks,
> >>  Daniel Dekany
> >>
> >>
>
> --
> Thanks,
>  Daniel Dekany
>
>

Re: The problem of "too human" text transformations

[Daniel Dekany <dd...@apache.org>]

Friday, June 8, 2018, 8:41:34 PM, Taher Alkhateeb wrote:

> One of the absolute hardest things for me to accomplish is to get the
> Arabic language with all the RTL quirks done right. I'm an Arab but even I
> suffer from the complexity of this language.

Speaking of which, I never hear about the needs of languages that are
very different (I guess) from the typical western European and US
languages, such as the Arabic, Chinese, Hindi and languages. I only
speak Hungarian and English (and has some very dim past experience
with Russian and German), so I have no idea if FM can be of any help
there.

> Hoping to achieve true and full internationalization is something that
> might not be very realistic IMHO. So maybe the best decision is not to
> support all languages but rather having the architecture and design of your
> templating engine done in an extinsible-friendly manner, which seems to be
> the case here?

Exactly.

I wonder how well the existing features work there, where we build on
the the localization features of the Java platform. I'm not sure how
well it works for languages that are very far from the "western
culture". (I had some feedback that suggests that for some such
country the defaults are wrong, but for a while it's possible to
handle that with custom number and date/time formats.)

> So yeah, perhaps I would prefer to go for English OOTB support and keep a
> flexible architecture in place.
>
> On Thu, Jun 7, 2018, 5:34 PM Daniel Dekany <dd...@apache.org> wrote:
>
>> Tuesday, June 5, 2018, 3:28:45 PM, Woonsan Ko wrote:
>>
>> > On Mon, Jun 4, 2018 at 5:04 AM, Daniel Dekany <dd...@apache.org>
>> wrote:
>> >> Sunday, June 3, 2018, 10:13:04 PM, David E Jones wrote:
>> >>
>> >>> You make good points Daniel. At a high level it reminds me of a
>> discussion
>> >>> with a new client a couple months ago on the topic of managing
>> templates
>> >>> for ecommerce sites. They want to use the Liquid template engine partly
>> >>> because they are doing custom ecommerce for some customers and Shopify
>> >>> (which uses Liquid, where Liquid came from) for others, and also
>> because
>> >>> Liquid is more constrained and easier for web developers/designers to
>> use.
>> >>>
>> >>> There is a case for template languages that are less developer
>> oriented and
>> >>> more designer oriented, or maybe better put because visual web
>> designers
>> >>> have no business mucking around with templates server side developer vs
>> >>> web/browser developer (even if this is stuff that generally runs on a
>> >>> server anyway).
>> >>>
>> >>> Right now, ie FM2, FreeMarker strikes a good balance between these (IMO
>> >>> anyway), and the built-ins are an important part of that. Maybe they
>> don't
>> >>> do every little thing everyone in the world might want but they cover
>> a lot
>> >>> of basics and common needs.
>> >>>
>> >>> At a lower level you are correct about template parsing and supporting
>> >>> arbitrary languages for inline expressions. The template parser still
>> needs
>> >>> to be able to figure out where the expression starts and stops. This
>> might
>> >>> not be too bad for C-like languages which current FM expressions are
>> (more
>> >>> or less, mostly the same demarcation characters and all), but is still
>> a
>> >>> challenge and could require something like surrounding expressions with
>> >>> parenthesis (along with nested parenthesis evaluation that FM already
>> does)
>> >>> and that's a little bit ugly.
>> >>
>> >> I wasn't even worried about that yet, but the semantical differences.
>> >> The expression and non-expression language (if they are separate at
>> >> all) build on a common mental model, so you can't, in general, cut out
>> >> the expression language from one language, and put it into another. So
>> >> I can only look at this on a case-by-case basis.
>> >>
>> >> As of the syntactical problems, at least if I write the expression
>> >> parser, I can just exit expression mode if the next token would lead
>> >> to an error, and let the "outer language" process the next character.
>> >> Like you have `a + b n=2`, then `n` would be an error, and the outer
>> >> parser will pick it up and recognize that as a parameter assignment.
>> >> What's tricky is making JavaCC to do this... it doesn't support
>> >> dynamic composing by design. ANTLR, I don't know (nor if it's fast
>> >> enough for use cases like a lot of ?eval/?interpret). For a hand
>> >> written parser, it's certainly not a problem. Though switching to hand
>> >> written is a questionable step, even if I personally find it tempting.
>> >>
>> >>> Anyway, sorry to derail the discussion.
>> >>>
>> >>> On the truncate built-in an optional String type second argument makes
>> >>> sense as a suffix to use if the string is truncated, ie that seems
>> like an
>> >>> easy thing to accommodate without reducing flexibility or having to
>> write
>> >>> that logic to do that inline in the template.
>> >>
>> >> There's also the question of where to cut the text. There's the
>> >> maximum length given as parameter (actually, you subtract the length
>> >> of the cut marker suffix), however if with applying that naively you
>> >> end up with white-space at the end, you probably want to trim that.
>> >> Also certainly if it ends with a ellipsis, dot or a sequence of dots,
>> >> and maybe even if with some other punctuation. There's also the
>> >> typical preference, over a certain maximum length, where you only cut
>> >> at the end of words (unless you have some extremely long word).
>> >
>> > Commons-text's WordUtils#abbreviate(str, lower, upper, appendToEnd)
>> > finds a space from the lower index to find where to cut.
>> > It doesn't care other white spaces nor ending dot(s), but one of
>> > existing simple algorithms at the moment. ;-)
>> >
>> >>
>> >>> On title casing it might be best to defer to what Java does (ie
>> >>> Character.toUpperCase() or .toTitleCase()), or were there issues with
>> that?
>> >>
>> >> They don't convert text to title case, only a single character. In the
>> >> most basic form of title case conversion of text, you convert the
>> >> first letter of each word to title case. You may already have some
>> >> tricky parts there with finding word boundaries. Then, in many
>> >> languages, like in English, you aren't supported to convert words like
>> >> "and", "of", etc. So it's obviously locale dependent.
>> >
>> > WordUtils#capitalizeFully() does. It doesn't recognize "and", "of", etc.
>> though.
>>
>> Both WordUtils methods seem to be too simplistic for real world usage,
>> to me at least. The default implementation should be actually usable
>> for at least English content, and beat typical quick in-house
>> solutions in quality. It seems that that's doable (see Jira issue).
>>
>> I guess the other languages will be left for the users (the algorithm
>> is pluggable per language). As far as I know the most common language
>> used for web content is still English, by far. So I guess it's the
>> language FM is used for the most as well. It's quite weird as English
>> isn't the most widely spoken language among the people who use the
>> Internet. But I'm afraid if we pick, let's say, the next 4 most common
>> web content languages, that can be be more offending for those left
>> out, than if we support English only, which people more or less have
>> get used to.
>>
>> > But as you said, if it can be customized through Configuration for
>> > locale specific use cases, I'd say it's perfect.
>> >
>> > Regards,
>> >
>> > Woonsan
>> >
>> >>
>> >>> -David
>> >>>
>> >>>
>> >>> On Thu, May 31, 2018 at 2:03 AM, Daniel Dekany <dd...@apache.org>
>> wrote:
>> >>>
>> >>>> Thursday, May 31, 2018, 1:16:26 AM, David E Jones wrote:
>> >>>>
>> >>>> > My general opinion on this sort of thing is there are lots of
>> libraries
>> >>>> and
>> >>>> > tools around in the Java world for doing string manipulations and
>> such,
>> >>>> > they don't need to be built into a template language. Taking that
>> one
>> >>>> step
>> >>>> > further, I'd actually prefer that the expression syntax in the
>> template
>> >>>> > language makes it easy to call an existing API vs trying to make
>> >>>> everything
>> >>>> > built into the template language.
>> >>>> >
>> >>>> > For example in addition to using FTL I also use Groovy heavily and
>> Groovy
>> >>>> > already has a great API including standard Java API augmentation
>> for all
>> >>>> > sorts of things. Yes, FTL could use the Groovy String and other
>> expanded
>> >>>> > objects as a reference for built in design and have a good set of
>> >>>> commonly
>> >>>> > needed things, but why not just defer to Groovy for those that use
>> it or
>> >>>> > whatever other API/library anyone might prefer.
>> >>>> >
>> >>>> > I know this is a VERY different direction from FM2. What I find most
>> >>>> useful
>> >>>> > in FTL is features like macros and the extensive directives and
>> other
>> >>>> > things that make a template language flexible and powerful. The
>> built-ins
>> >>>> > are handy sometimes, but often don't do what I need anyway so other
>> than
>> >>>> > things like ?has_content I don't use them a lot.
>> >>>>
>> >>>> An important (and as far as I'm concerned, the main) reason for many
>> >>>> built-ins is/was that frameworks and other products drop in FreeMarker
>> >>>> for templates and then often don't give much attention to give tools
>> >>>> to the template authors. Then the poor people who actually has to
>> >>>> create templates come and ask, "I must do X... how can I do that in
>> >>>> FreeMaker?". The less often we have to say "Please pester the
>> >>>> developers (who are often from another company, or are on another
>> >>>> sub-project by now => you have near 0 chance) to expose such
>> >>>> functionality to the templates", the better. Instead we can say "Like
>> >>>> foo?x", and furthermore, ?x is documented, and people can actually
>> >>>> find its documentation, even years later. Surely if you have some
>> >>>> small company with the sharp people, this is not a problem for you.
>> >>>> Actually, then you very often don't want the one-size-fits-all
>> >>>> solutions anyway.
>> >>>>
>> >>>> > I've mentioned this before but what would be a killer feature for
>> FTL is
>> >>>> to
>> >>>> > support pluggable expression evaluators. The FTL expression syntax
>> isn't
>> >>>> > awful
>> >>>>
>> >>>> (Yes it is... but that's besides the point until I have fixed it in
>> >>>> FM3.)
>> >>>>
>> >>>> > but is not nearly as convenient as Groovy, especially when you are
>> >>>> > using it extensively elsewhere.
>> >>>>
>> >>>> The theoretical problem is of course that both languages (the "outer"
>> >>>> template language and the language from where you rip off the
>> >>>> expression language from) have its own ideas, and so there will be
>> >>>> mismatches. The expression language in a language and other parts of
>> >>>> the language aren't separate in general anyway. Which leads to the
>> >>>> question that if you are using Groovy expressions, why not also use
>> >>>> for loops and whatnot from Groovy? (This is fairly common way of
>> >>>> creating template language, just taking an existing language, and add
>> >>>> some convenience syntax to embed it into the static text.) So then we
>> >>>> don't need our own #list, #if, etc. implementation either. But then,
>> >>>> the original idea was that a template language is a domain specific
>> >>>> language, and as such can have more specialized, say, loop constructs
>> >>>> for that domain. (I'm not saying that FM2 did great there, but there
>> >>>> was a lack of determination there as well.) If that's not the case
>> >>>> though...
>> >>>>
>> >>>> Anyway, when and if FM3 will be more mature, we can see if how doable
>> >>>> this is in reality. It's also a question if we are talking about a
>> >>>> syntax, that in 99% looks like Groovy, or we are talking about actual
>> >>>> Groovy. In the last case TemplateModel-s and the particular way null
>> >>>> handling will work will be an obstacle for sure. Not to mention
>> >>>> TemplateMarkupOutputModel-s, that AFAIK don't even have commonly used
>> >>>> equivalent (like a class in the standard Java API).
>> >>>>
>> >>>> > If the default FTL expressions were not used you couldn't use
>> >>>> > built-ins and instead would just have expressions that are expected
>> >>>> > to evaluate to a boolean, string, or arbitrary object and can be
>> >>>> > whatever.
>> >>>>
>> >>>> At least in FM3 built-ins will be just functions you have defined in
>> >>>> the "core" FTL namespace (whose content is specified in template
>> >>>> language Dialect you use) that's "statically linked" to the templates
>> >>>> (allowing some parse-time checks and somewhat faster calls). So
>> >>>> `foo?bar(1)` is basically just a shorthand syntax for `<#import core =
>> >>>> 'freemarker://dialect-or-something'>` and `core.bar(foo, 1)`. There's
>> >>>> a standard/default Dialect of course, which has the big advantage that
>> >>>> it's documented in the official Manual, is used in SO answers, etc.
>> >>>>
>> >>>> > How important is this? Important enough that I've
>> >>>> > considered using a different template language. If FTL wasn't so
>> >>>> > darn convenience for XML transformation using macros I'd probably be
>> >>>> > using Groovy GString templates instead, and just for the far better
>> >>>> > syntax in Groovy.
>> >>>> >
>> >>>> > This also goes back to the object wrapping discussion which is
>> another
>> >>>> can
>> >>>> > of worms...
>> >>>>
>> >>>> Yeah, it's one of the things that I really wanted to get rid of, but
>> >>>> it's a very disrupting transition to do. I don't mean for most of the
>> >>>> users, but for the source code. On the brighter side, what really
>> >>>> happens when you do that transition is that instead of something like:
>> >>>>
>> >>>>   if (value is TemplateMapLikeModel) {
>> >>>>       ... ((TemplateMapLikeModel) value).get(k) ...
>> >>>>   }
>> >>>>
>> >>>> you, in first approach, rewrite it to something like:
>> >>>>
>> >>>>   if (mop.isMapLike(value)) {
>> >>>>       ... mop.get(value, k) ...
>> >>>>   }
>> >>>>
>> >>>> Of course that would be a silly MOP API design, so this below is more
>> >>>> likely, but now it's visible that one comes from the other trivially:
>> >>>>
>> >>>>
>> >>>>   MapLike m = mop.tryGetAsMapLike(value);
>> >>>>   if (m != null) {
>> >>>>       ... m.get(k) ...
>> >>>>   }
>> >>>>
>> >>>> Hence when we have our TemplateModel-s cleaned up, it's easier to
>> >>>> migrate to the MOP approach. The "type system" (just a mental model
>> >>>> really) depicted be the TemplateModel still exists, you just use a MOP
>> >>>> to implement it.
>> >>>>
>> >>>> > I don't mean to derail this discussion, but especially these days
>> in a
>> >>>> > world of a huge number of great Java libraries out there it is often
>> >>>> better
>> >>>> > to not try to do everything and instead focus on core strengths
>> that you
>> >>>> > can't get anywhere else, and make it easy for people to use other
>> open
>> >>>> > source projects for their core strengths. This could even go as far
>> as
>> >>>> > FreeMarker builds that have dependencies on certain commonly used
>> open
>> >>>> > source projects, there are many even in the ASF that do various
>> things
>> >>>> that
>> >>>> > FTL built-ins do but with far more options and well known behavior
>> for
>> >>>> the
>> >>>> > many who use them (like various Commons projects).
>> >>>>
>> >>>> It's really the duty of the Dialects feature of FM3. You can put
>> >>>> together your "toolbox" from whatever you want, like call Apache
>> >>>> Commons. I mean it's unrelated to question of expression languages, as
>> >>>> far as I see. (Like, you can call plain Java methods from FM2 as
>> >>>> well.)
>> >>>>
>> >>>> If we want to use Apache Commons etc. in implementing the functions of
>> >>>> the standard dialect... I *guess* we won't benefit from it much at
>> >>>> this point, but I need to see it case-by-case to tell, like, the FM
>> >>>> implementation of function x sucks, while Apache Commons StringUtils.x
>> >>>> is great, etc. In any case, users don't care, as far as they can call
>> >>>> ?x and it does its job. My point in this thread is that we should be
>> >>>> able to tell users to "just write ?x" (see earlier), instead of add
>> >>>> your own solution somehow. (Also, if ?x works, they prefer if we have
>> >>>> less dependencies. Especially if we want to support Andorid.)
>> >>>>
>> >>>> > Would supporting different expression languages cause huge problems
>> for
>> >>>> IDE
>> >>>> > support? Possibly, but IDEs these days have lots of support for
>> language
>> >>>> > injection and layers that are meant for just this sort of thing.
>> >>>>
>> >>>> At least looking from big enough distance it's easy. :) Note that so
>> >>>> far we fail to deliver an editor for Eclipse that integrates the
>> >>>> existing HTML editor functionality with FTL... :-> Please everyone go
>> >>>> and help out Angelo Zerr so that this will change:
>> >>>>
>> >>>> https://github.com/angelozerr/lsp4e-freemarker
>> >>>> https://github.com/angelozerr/freemarker-languageserver
>> >>>>
>> >>>> Anyway, IDE support won't be the obstacle. At worse only the default
>> >>>> expression syntax will have reasonable IDE support.
>> >>>>
>> >>>> > In theory FreeMarker itself could be a heck of a lot smaller and
>> >>>> > simpler and at the same time be far more flexible and powerful. IMO
>> >>>> > that could be a game changer in the world of Java-based template
>> >>>> > languages. Some of this would be more work than not supporting this
>> >>>> > sort of flexibility (ie support Groovy syntax as an option, ie
>> >>>> > without requiring the much larger Groovy jar files as dependencies
>> >>>> > by having a default built-in expression syntax), but by simplifying
>> >>>> > other things it would save a heck of a lot of work (including design
>> >>>> > work).
>> >>>>
>> >>>> So, I'm not sure I follow what exactly are you proposing. To use
>> >>>> vanilla Groovy (the actual implementation) for expressions? How does
>> >>>> that blend with the directives (which mostly also have similar pairs
>> >>>> in Groovy), and other (planned) FM features? (And... why's the result
>> >>>> called FreeMarker at all? ;) )
>> >>>>
>> >>>> > -David
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <
>> >>>> slidingfilaments@gmail.com
>> >>>> >> wrote:
>> >>>> >
>> >>>> >> I'm not a subject matter expert here, but I always thought sensible
>> >>>> >> defaults with the ability to override using configuration is a wise
>> >>>> >> move because you appeal to many people out of the box and then
>> allow
>> >>>> >> minority to also not feel locked out when they need to override.
>> Also,
>> >>>> >> as a general rule, I found from the projects we worked on that
>> >>>> >> internationalization is always a challenge, but you can also
>> resolve
>> >>>> >> that with the same solution (sensible defaults with configuration
>> >>>> >> overrides); then you simply let the community adopt and improve the
>> >>>> >> code for non-English parts.
>> >>>> >>
>> >>>> >> Of course then the challenge becomes, what's a sensible default? I
>> >>>> >> wish I had a dollar every time I asked that! I'd favor simplicity
>> >>>> >> where possible. In your example above maybe "No space f..." is the
>> >>>> >> simplest, and then people can get fancy if they want to?
>> >>>> >>
>> >>>> >> My 2 cents, sorry for the noise :)
>> >>>> >>
>> >>>> >> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <
>> ddekany@apache.org>
>> >>>> >> wrote:
>> >>>> >> > There's this old frequently requested built-in,
>> >>>> >> > maybeTooLongText?truncate(n). Like "No space for such a long
>> >>>> >> > title"?truncate(13) would give "No space...". The problem with
>> these
>> >>>> >> > is that there's no single algorithm for this that fits the need
>> of
>> >>>> >> > everyone. Like maybe the result should be "No space f...", or "No
>> >>>> >> > space [...]", etc. This can even change depending on the current
>> value
>> >>>> >> > of the `locale` FreeMarker setting.
>> >>>> >> >
>> >>>> >> > Now there's another similar wish, for "?title_case", which has
>> the
>> >>>> >> > same problem, plus it's much more locale dependent. See:
>> >>>> >> >
>> >>>> >> > https://issues.apache.org/jira/browse/FREEMARKER-97
>> >>>> >> >
>> >>>> >> > We do provide locale dependent functions, like number and date
>> >>>> >> > formatting. This is pretty much an FM tradition, to treat
>> non-English
>> >>>> >> > users as equals (well, except that the keywords aren't
>> localizable).
>> >>>> >> > However, we push those tasks to the Java platform. But it doesn't
>> >>>> >> > support title case (as far as I know). Besides there can be
>> different
>> >>>> >> > preferences even inside the same country, just as with ?truncate.
>> >>>> >> >
>> >>>> >> > So, it's acceptable to chicken out ;) saying that none of the
>> above
>> >>>> >> > has a widely accepted default, so the correct solution is
>> letting the
>> >>>> >> > user solve this. But I think in practice what happens is that
>> people
>> >>>> >> > need the functionality (especially ?truncate), and then has to
>> come up
>> >>>> >> > with some ad-hoc solution on the spot, which will be almost
>> always
>> >>>> >> > worse than the default we could provide. So we might as well just
>> >>>> >> > provide some decent default, sparing time for lot of users.
>> Then, then
>> >>>> >> > if they are particular about the algorithm (which is certainly
>> rare
>> >>>> >> > nowadays), let them plug their own via the Configurable API. So
>> there
>> >>>> >> > would be something like
>> >>>> >> > cfg.setTruncateImplementation(localToImplementationMap), where
>> the
>> >>>> >> > parameter maps locale-s to Java functions and defines a fallback
>> >>>> >> > function. (I haven't worked out the API details, as you can see.)
>> >>>> >> >
>> >>>> >> > What do you think?
>> >>>> >> >
>> >>>> >> > --
>> >>>> >> > Thanks,
>> >>>> >> >  Daniel Dekany
>> >>>> >> >
>> >>>> >>
>> >>>>
>> >>>> --
>> >>>> Thanks,
>> >>>>  Daniel Dekany
>> >>>>
>> >>>>
>> >>
>> >> --
>> >> Thanks,
>> >>  Daniel Dekany
>> >>
>> >
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>

-- 
Thanks,
 Daniel Dekany

Re: The problem of "too human" text transformations

[Taher Alkhateeb <sl...@gmail.com>]

One of the absolute hardest things for me to accomplish is to get the
Arabic language with all the RTL quirks done right. I'm an Arab but even I
suffer from the complexity of this language.

Hoping to achieve true and full internationalization is something that
might not be very realistic IMHO. So maybe the best decision is not to
support all languages but rather having the architecture and design of your
templating engine done in an extinsible-friendly manner, which seems to be
the case here?

So yeah, perhaps I would prefer to go for English OOTB support and keep a
flexible architecture in place.

On Thu, Jun 7, 2018, 5:34 PM Daniel Dekany <dd...@apache.org> wrote:

> Tuesday, June 5, 2018, 3:28:45 PM, Woonsan Ko wrote:
>
> > On Mon, Jun 4, 2018 at 5:04 AM, Daniel Dekany <dd...@apache.org>
> wrote:
> >> Sunday, June 3, 2018, 10:13:04 PM, David E Jones wrote:
> >>
> >>> You make good points Daniel. At a high level it reminds me of a
> discussion
> >>> with a new client a couple months ago on the topic of managing
> templates
> >>> for ecommerce sites. They want to use the Liquid template engine partly
> >>> because they are doing custom ecommerce for some customers and Shopify
> >>> (which uses Liquid, where Liquid came from) for others, and also
> because
> >>> Liquid is more constrained and easier for web developers/designers to
> use.
> >>>
> >>> There is a case for template languages that are less developer
> oriented and
> >>> more designer oriented, or maybe better put because visual web
> designers
> >>> have no business mucking around with templates server side developer vs
> >>> web/browser developer (even if this is stuff that generally runs on a
> >>> server anyway).
> >>>
> >>> Right now, ie FM2, FreeMarker strikes a good balance between these (IMO
> >>> anyway), and the built-ins are an important part of that. Maybe they
> don't
> >>> do every little thing everyone in the world might want but they cover
> a lot
> >>> of basics and common needs.
> >>>
> >>> At a lower level you are correct about template parsing and supporting
> >>> arbitrary languages for inline expressions. The template parser still
> needs
> >>> to be able to figure out where the expression starts and stops. This
> might
> >>> not be too bad for C-like languages which current FM expressions are
> (more
> >>> or less, mostly the same demarcation characters and all), but is still
> a
> >>> challenge and could require something like surrounding expressions with
> >>> parenthesis (along with nested parenthesis evaluation that FM already
> does)
> >>> and that's a little bit ugly.
> >>
> >> I wasn't even worried about that yet, but the semantical differences.
> >> The expression and non-expression language (if they are separate at
> >> all) build on a common mental model, so you can't, in general, cut out
> >> the expression language from one language, and put it into another. So
> >> I can only look at this on a case-by-case basis.
> >>
> >> As of the syntactical problems, at least if I write the expression
> >> parser, I can just exit expression mode if the next token would lead
> >> to an error, and let the "outer language" process the next character.
> >> Like you have `a + b n=2`, then `n` would be an error, and the outer
> >> parser will pick it up and recognize that as a parameter assignment.
> >> What's tricky is making JavaCC to do this... it doesn't support
> >> dynamic composing by design. ANTLR, I don't know (nor if it's fast
> >> enough for use cases like a lot of ?eval/?interpret). For a hand
> >> written parser, it's certainly not a problem. Though switching to hand
> >> written is a questionable step, even if I personally find it tempting.
> >>
> >>> Anyway, sorry to derail the discussion.
> >>>
> >>> On the truncate built-in an optional String type second argument makes
> >>> sense as a suffix to use if the string is truncated, ie that seems
> like an
> >>> easy thing to accommodate without reducing flexibility or having to
> write
> >>> that logic to do that inline in the template.
> >>
> >> There's also the question of where to cut the text. There's the
> >> maximum length given as parameter (actually, you subtract the length
> >> of the cut marker suffix), however if with applying that naively you
> >> end up with white-space at the end, you probably want to trim that.
> >> Also certainly if it ends with a ellipsis, dot or a sequence of dots,
> >> and maybe even if with some other punctuation. There's also the
> >> typical preference, over a certain maximum length, where you only cut
> >> at the end of words (unless you have some extremely long word).
> >
> > Commons-text's WordUtils#abbreviate(str, lower, upper, appendToEnd)
> > finds a space from the lower index to find where to cut.
> > It doesn't care other white spaces nor ending dot(s), but one of
> > existing simple algorithms at the moment. ;-)
> >
> >>
> >>> On title casing it might be best to defer to what Java does (ie
> >>> Character.toUpperCase() or .toTitleCase()), or were there issues with
> that?
> >>
> >> They don't convert text to title case, only a single character. In the
> >> most basic form of title case conversion of text, you convert the
> >> first letter of each word to title case. You may already have some
> >> tricky parts there with finding word boundaries. Then, in many
> >> languages, like in English, you aren't supported to convert words like
> >> "and", "of", etc. So it's obviously locale dependent.
> >
> > WordUtils#capitalizeFully() does. It doesn't recognize "and", "of", etc.
> though.
>
> Both WordUtils methods seem to be too simplistic for real world usage,
> to me at least. The default implementation should be actually usable
> for at least English content, and beat typical quick in-house
> solutions in quality. It seems that that's doable (see Jira issue).
>
> I guess the other languages will be left for the users (the algorithm
> is pluggable per language). As far as I know the most common language
> used for web content is still English, by far. So I guess it's the
> language FM is used for the most as well. It's quite weird as English
> isn't the most widely spoken language among the people who use the
> Internet. But I'm afraid if we pick, let's say, the next 4 most common
> web content languages, that can be be more offending for those left
> out, than if we support English only, which people more or less have
> get used to.
>
> > But as you said, if it can be customized through Configuration for
> > locale specific use cases, I'd say it's perfect.
> >
> > Regards,
> >
> > Woonsan
> >
> >>
> >>> -David
> >>>
> >>>
> >>> On Thu, May 31, 2018 at 2:03 AM, Daniel Dekany <dd...@apache.org>
> wrote:
> >>>
> >>>> Thursday, May 31, 2018, 1:16:26 AM, David E Jones wrote:
> >>>>
> >>>> > My general opinion on this sort of thing is there are lots of
> libraries
> >>>> and
> >>>> > tools around in the Java world for doing string manipulations and
> such,
> >>>> > they don't need to be built into a template language. Taking that
> one
> >>>> step
> >>>> > further, I'd actually prefer that the expression syntax in the
> template
> >>>> > language makes it easy to call an existing API vs trying to make
> >>>> everything
> >>>> > built into the template language.
> >>>> >
> >>>> > For example in addition to using FTL I also use Groovy heavily and
> Groovy
> >>>> > already has a great API including standard Java API augmentation
> for all
> >>>> > sorts of things. Yes, FTL could use the Groovy String and other
> expanded
> >>>> > objects as a reference for built in design and have a good set of
> >>>> commonly
> >>>> > needed things, but why not just defer to Groovy for those that use
> it or
> >>>> > whatever other API/library anyone might prefer.
> >>>> >
> >>>> > I know this is a VERY different direction from FM2. What I find most
> >>>> useful
> >>>> > in FTL is features like macros and the extensive directives and
> other
> >>>> > things that make a template language flexible and powerful. The
> built-ins
> >>>> > are handy sometimes, but often don't do what I need anyway so other
> than
> >>>> > things like ?has_content I don't use them a lot.
> >>>>
> >>>> An important (and as far as I'm concerned, the main) reason for many
> >>>> built-ins is/was that frameworks and other products drop in FreeMarker
> >>>> for templates and then often don't give much attention to give tools
> >>>> to the template authors. Then the poor people who actually has to
> >>>> create templates come and ask, "I must do X... how can I do that in
> >>>> FreeMaker?". The less often we have to say "Please pester the
> >>>> developers (who are often from another company, or are on another
> >>>> sub-project by now => you have near 0 chance) to expose such
> >>>> functionality to the templates", the better. Instead we can say "Like
> >>>> foo?x", and furthermore, ?x is documented, and people can actually
> >>>> find its documentation, even years later. Surely if you have some
> >>>> small company with the sharp people, this is not a problem for you.
> >>>> Actually, then you very often don't want the one-size-fits-all
> >>>> solutions anyway.
> >>>>
> >>>> > I've mentioned this before but what would be a killer feature for
> FTL is
> >>>> to
> >>>> > support pluggable expression evaluators. The FTL expression syntax
> isn't
> >>>> > awful
> >>>>
> >>>> (Yes it is... but that's besides the point until I have fixed it in
> >>>> FM3.)
> >>>>
> >>>> > but is not nearly as convenient as Groovy, especially when you are
> >>>> > using it extensively elsewhere.
> >>>>
> >>>> The theoretical problem is of course that both languages (the "outer"
> >>>> template language and the language from where you rip off the
> >>>> expression language from) have its own ideas, and so there will be
> >>>> mismatches. The expression language in a language and other parts of
> >>>> the language aren't separate in general anyway. Which leads to the
> >>>> question that if you are using Groovy expressions, why not also use
> >>>> for loops and whatnot from Groovy? (This is fairly common way of
> >>>> creating template language, just taking an existing language, and add
> >>>> some convenience syntax to embed it into the static text.) So then we
> >>>> don't need our own #list, #if, etc. implementation either. But then,
> >>>> the original idea was that a template language is a domain specific
> >>>> language, and as such can have more specialized, say, loop constructs
> >>>> for that domain. (I'm not saying that FM2 did great there, but there
> >>>> was a lack of determination there as well.) If that's not the case
> >>>> though...
> >>>>
> >>>> Anyway, when and if FM3 will be more mature, we can see if how doable
> >>>> this is in reality. It's also a question if we are talking about a
> >>>> syntax, that in 99% looks like Groovy, or we are talking about actual
> >>>> Groovy. In the last case TemplateModel-s and the particular way null
> >>>> handling will work will be an obstacle for sure. Not to mention
> >>>> TemplateMarkupOutputModel-s, that AFAIK don't even have commonly used
> >>>> equivalent (like a class in the standard Java API).
> >>>>
> >>>> > If the default FTL expressions were not used you couldn't use
> >>>> > built-ins and instead would just have expressions that are expected
> >>>> > to evaluate to a boolean, string, or arbitrary object and can be
> >>>> > whatever.
> >>>>
> >>>> At least in FM3 built-ins will be just functions you have defined in
> >>>> the "core" FTL namespace (whose content is specified in template
> >>>> language Dialect you use) that's "statically linked" to the templates
> >>>> (allowing some parse-time checks and somewhat faster calls). So
> >>>> `foo?bar(1)` is basically just a shorthand syntax for `<#import core =
> >>>> 'freemarker://dialect-or-something'>` and `core.bar(foo, 1)`. There's
> >>>> a standard/default Dialect of course, which has the big advantage that
> >>>> it's documented in the official Manual, is used in SO answers, etc.
> >>>>
> >>>> > How important is this? Important enough that I've
> >>>> > considered using a different template language. If FTL wasn't so
> >>>> > darn convenience for XML transformation using macros I'd probably be
> >>>> > using Groovy GString templates instead, and just for the far better
> >>>> > syntax in Groovy.
> >>>> >
> >>>> > This also goes back to the object wrapping discussion which is
> another
> >>>> can
> >>>> > of worms...
> >>>>
> >>>> Yeah, it's one of the things that I really wanted to get rid of, but
> >>>> it's a very disrupting transition to do. I don't mean for most of the
> >>>> users, but for the source code. On the brighter side, what really
> >>>> happens when you do that transition is that instead of something like:
> >>>>
> >>>>   if (value is TemplateMapLikeModel) {
> >>>>       ... ((TemplateMapLikeModel) value).get(k) ...
> >>>>   }
> >>>>
> >>>> you, in first approach, rewrite it to something like:
> >>>>
> >>>>   if (mop.isMapLike(value)) {
> >>>>       ... mop.get(value, k) ...
> >>>>   }
> >>>>
> >>>> Of course that would be a silly MOP API design, so this below is more
> >>>> likely, but now it's visible that one comes from the other trivially:
> >>>>
> >>>>
> >>>>   MapLike m = mop.tryGetAsMapLike(value);
> >>>>   if (m != null) {
> >>>>       ... m.get(k) ...
> >>>>   }
> >>>>
> >>>> Hence when we have our TemplateModel-s cleaned up, it's easier to
> >>>> migrate to the MOP approach. The "type system" (just a mental model
> >>>> really) depicted be the TemplateModel still exists, you just use a MOP
> >>>> to implement it.
> >>>>
> >>>> > I don't mean to derail this discussion, but especially these days
> in a
> >>>> > world of a huge number of great Java libraries out there it is often
> >>>> better
> >>>> > to not try to do everything and instead focus on core strengths
> that you
> >>>> > can't get anywhere else, and make it easy for people to use other
> open
> >>>> > source projects for their core strengths. This could even go as far
> as
> >>>> > FreeMarker builds that have dependencies on certain commonly used
> open
> >>>> > source projects, there are many even in the ASF that do various
> things
> >>>> that
> >>>> > FTL built-ins do but with far more options and well known behavior
> for
> >>>> the
> >>>> > many who use them (like various Commons projects).
> >>>>
> >>>> It's really the duty of the Dialects feature of FM3. You can put
> >>>> together your "toolbox" from whatever you want, like call Apache
> >>>> Commons. I mean it's unrelated to question of expression languages, as
> >>>> far as I see. (Like, you can call plain Java methods from FM2 as
> >>>> well.)
> >>>>
> >>>> If we want to use Apache Commons etc. in implementing the functions of
> >>>> the standard dialect... I *guess* we won't benefit from it much at
> >>>> this point, but I need to see it case-by-case to tell, like, the FM
> >>>> implementation of function x sucks, while Apache Commons StringUtils.x
> >>>> is great, etc. In any case, users don't care, as far as they can call
> >>>> ?x and it does its job. My point in this thread is that we should be
> >>>> able to tell users to "just write ?x" (see earlier), instead of add
> >>>> your own solution somehow. (Also, if ?x works, they prefer if we have
> >>>> less dependencies. Especially if we want to support Andorid.)
> >>>>
> >>>> > Would supporting different expression languages cause huge problems
> for
> >>>> IDE
> >>>> > support? Possibly, but IDEs these days have lots of support for
> language
> >>>> > injection and layers that are meant for just this sort of thing.
> >>>>
> >>>> At least looking from big enough distance it's easy. :) Note that so
> >>>> far we fail to deliver an editor for Eclipse that integrates the
> >>>> existing HTML editor functionality with FTL... :-> Please everyone go
> >>>> and help out Angelo Zerr so that this will change:
> >>>>
> >>>> https://github.com/angelozerr/lsp4e-freemarker
> >>>> https://github.com/angelozerr/freemarker-languageserver
> >>>>
> >>>> Anyway, IDE support won't be the obstacle. At worse only the default
> >>>> expression syntax will have reasonable IDE support.
> >>>>
> >>>> > In theory FreeMarker itself could be a heck of a lot smaller and
> >>>> > simpler and at the same time be far more flexible and powerful. IMO
> >>>> > that could be a game changer in the world of Java-based template
> >>>> > languages. Some of this would be more work than not supporting this
> >>>> > sort of flexibility (ie support Groovy syntax as an option, ie
> >>>> > without requiring the much larger Groovy jar files as dependencies
> >>>> > by having a default built-in expression syntax), but by simplifying
> >>>> > other things it would save a heck of a lot of work (including design
> >>>> > work).
> >>>>
> >>>> So, I'm not sure I follow what exactly are you proposing. To use
> >>>> vanilla Groovy (the actual implementation) for expressions? How does
> >>>> that blend with the directives (which mostly also have similar pairs
> >>>> in Groovy), and other (planned) FM features? (And... why's the result
> >>>> called FreeMarker at all? ;) )
> >>>>
> >>>> > -David
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <
> >>>> slidingfilaments@gmail.com
> >>>> >> wrote:
> >>>> >
> >>>> >> I'm not a subject matter expert here, but I always thought sensible
> >>>> >> defaults with the ability to override using configuration is a wise
> >>>> >> move because you appeal to many people out of the box and then
> allow
> >>>> >> minority to also not feel locked out when they need to override.
> Also,
> >>>> >> as a general rule, I found from the projects we worked on that
> >>>> >> internationalization is always a challenge, but you can also
> resolve
> >>>> >> that with the same solution (sensible defaults with configuration
> >>>> >> overrides); then you simply let the community adopt and improve the
> >>>> >> code for non-English parts.
> >>>> >>
> >>>> >> Of course then the challenge becomes, what's a sensible default? I
> >>>> >> wish I had a dollar every time I asked that! I'd favor simplicity
> >>>> >> where possible. In your example above maybe "No space f..." is the
> >>>> >> simplest, and then people can get fancy if they want to?
> >>>> >>
> >>>> >> My 2 cents, sorry for the noise :)
> >>>> >>
> >>>> >> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <
> ddekany@apache.org>
> >>>> >> wrote:
> >>>> >> > There's this old frequently requested built-in,
> >>>> >> > maybeTooLongText?truncate(n). Like "No space for such a long
> >>>> >> > title"?truncate(13) would give "No space...". The problem with
> these
> >>>> >> > is that there's no single algorithm for this that fits the need
> of
> >>>> >> > everyone. Like maybe the result should be "No space f...", or "No
> >>>> >> > space [...]", etc. This can even change depending on the current
> value
> >>>> >> > of the `locale` FreeMarker setting.
> >>>> >> >
> >>>> >> > Now there's another similar wish, for "?title_case", which has
> the
> >>>> >> > same problem, plus it's much more locale dependent. See:
> >>>> >> >
> >>>> >> > https://issues.apache.org/jira/browse/FREEMARKER-97
> >>>> >> >
> >>>> >> > We do provide locale dependent functions, like number and date
> >>>> >> > formatting. This is pretty much an FM tradition, to treat
> non-English
> >>>> >> > users as equals (well, except that the keywords aren't
> localizable).
> >>>> >> > However, we push those tasks to the Java platform. But it doesn't
> >>>> >> > support title case (as far as I know). Besides there can be
> different
> >>>> >> > preferences even inside the same country, just as with ?truncate.
> >>>> >> >
> >>>> >> > So, it's acceptable to chicken out ;) saying that none of the
> above
> >>>> >> > has a widely accepted default, so the correct solution is
> letting the
> >>>> >> > user solve this. But I think in practice what happens is that
> people
> >>>> >> > need the functionality (especially ?truncate), and then has to
> come up
> >>>> >> > with some ad-hoc solution on the spot, which will be almost
> always
> >>>> >> > worse than the default we could provide. So we might as well just
> >>>> >> > provide some decent default, sparing time for lot of users.
> Then, then
> >>>> >> > if they are particular about the algorithm (which is certainly
> rare
> >>>> >> > nowadays), let them plug their own via the Configurable API. So
> there
> >>>> >> > would be something like
> >>>> >> > cfg.setTruncateImplementation(localToImplementationMap), where
> the
> >>>> >> > parameter maps locale-s to Java functions and defines a fallback
> >>>> >> > function. (I haven't worked out the API details, as you can see.)
> >>>> >> >
> >>>> >> > What do you think?
> >>>> >> >
> >>>> >> > --
> >>>> >> > Thanks,
> >>>> >> >  Daniel Dekany
> >>>> >> >
> >>>> >>
> >>>>
> >>>> --
> >>>> Thanks,
> >>>>  Daniel Dekany
> >>>>
> >>>>
> >>
> >> --
> >> Thanks,
> >>  Daniel Dekany
> >>
> >
>
> --
> Thanks,
>  Daniel Dekany
>
>

Re: The problem of "too human" text transformations

[Daniel Dekany <dd...@apache.org>]

Tuesday, June 5, 2018, 3:28:45 PM, Woonsan Ko wrote:

> On Mon, Jun 4, 2018 at 5:04 AM, Daniel Dekany <dd...@apache.org> wrote:
>> Sunday, June 3, 2018, 10:13:04 PM, David E Jones wrote:
>>
>>> You make good points Daniel. At a high level it reminds me of a discussion
>>> with a new client a couple months ago on the topic of managing templates
>>> for ecommerce sites. They want to use the Liquid template engine partly
>>> because they are doing custom ecommerce for some customers and Shopify
>>> (which uses Liquid, where Liquid came from) for others, and also because
>>> Liquid is more constrained and easier for web developers/designers to use.
>>>
>>> There is a case for template languages that are less developer oriented and
>>> more designer oriented, or maybe better put because visual web designers
>>> have no business mucking around with templates server side developer vs
>>> web/browser developer (even if this is stuff that generally runs on a
>>> server anyway).
>>>
>>> Right now, ie FM2, FreeMarker strikes a good balance between these (IMO
>>> anyway), and the built-ins are an important part of that. Maybe they don't
>>> do every little thing everyone in the world might want but they cover a lot
>>> of basics and common needs.
>>>
>>> At a lower level you are correct about template parsing and supporting
>>> arbitrary languages for inline expressions. The template parser still needs
>>> to be able to figure out where the expression starts and stops. This might
>>> not be too bad for C-like languages which current FM expressions are (more
>>> or less, mostly the same demarcation characters and all), but is still a
>>> challenge and could require something like surrounding expressions with
>>> parenthesis (along with nested parenthesis evaluation that FM already does)
>>> and that's a little bit ugly.
>>
>> I wasn't even worried about that yet, but the semantical differences.
>> The expression and non-expression language (if they are separate at
>> all) build on a common mental model, so you can't, in general, cut out
>> the expression language from one language, and put it into another. So
>> I can only look at this on a case-by-case basis.
>>
>> As of the syntactical problems, at least if I write the expression
>> parser, I can just exit expression mode if the next token would lead
>> to an error, and let the "outer language" process the next character.
>> Like you have `a + b n=2`, then `n` would be an error, and the outer
>> parser will pick it up and recognize that as a parameter assignment.
>> What's tricky is making JavaCC to do this... it doesn't support
>> dynamic composing by design. ANTLR, I don't know (nor if it's fast
>> enough for use cases like a lot of ?eval/?interpret). For a hand
>> written parser, it's certainly not a problem. Though switching to hand
>> written is a questionable step, even if I personally find it tempting.
>>
>>> Anyway, sorry to derail the discussion.
>>>
>>> On the truncate built-in an optional String type second argument makes
>>> sense as a suffix to use if the string is truncated, ie that seems like an
>>> easy thing to accommodate without reducing flexibility or having to write
>>> that logic to do that inline in the template.
>>
>> There's also the question of where to cut the text. There's the
>> maximum length given as parameter (actually, you subtract the length
>> of the cut marker suffix), however if with applying that naively you
>> end up with white-space at the end, you probably want to trim that.
>> Also certainly if it ends with a ellipsis, dot or a sequence of dots,
>> and maybe even if with some other punctuation. There's also the
>> typical preference, over a certain maximum length, where you only cut
>> at the end of words (unless you have some extremely long word).
>
> Commons-text's WordUtils#abbreviate(str, lower, upper, appendToEnd)
> finds a space from the lower index to find where to cut.
> It doesn't care other white spaces nor ending dot(s), but one of
> existing simple algorithms at the moment. ;-)
>
>>
>>> On title casing it might be best to defer to what Java does (ie
>>> Character.toUpperCase() or .toTitleCase()), or were there issues with that?
>>
>> They don't convert text to title case, only a single character. In the
>> most basic form of title case conversion of text, you convert the
>> first letter of each word to title case. You may already have some
>> tricky parts there with finding word boundaries. Then, in many
>> languages, like in English, you aren't supported to convert words like
>> "and", "of", etc. So it's obviously locale dependent.
>
> WordUtils#capitalizeFully() does. It doesn't recognize "and", "of", etc. though.

Both WordUtils methods seem to be too simplistic for real world usage,
to me at least. The default implementation should be actually usable
for at least English content, and beat typical quick in-house
solutions in quality. It seems that that's doable (see Jira issue).

I guess the other languages will be left for the users (the algorithm
is pluggable per language). As far as I know the most common language
used for web content is still English, by far. So I guess it's the
language FM is used for the most as well. It's quite weird as English
isn't the most widely spoken language among the people who use the
Internet. But I'm afraid if we pick, let's say, the next 4 most common
web content languages, that can be be more offending for those left
out, than if we support English only, which people more or less have
get used to.

> But as you said, if it can be customized through Configuration for
> locale specific use cases, I'd say it's perfect.
>
> Regards,
>
> Woonsan
>
>>
>>> -David
>>>
>>>
>>> On Thu, May 31, 2018 at 2:03 AM, Daniel Dekany <dd...@apache.org> wrote:
>>>
>>>> Thursday, May 31, 2018, 1:16:26 AM, David E Jones wrote:
>>>>
>>>> > My general opinion on this sort of thing is there are lots of libraries
>>>> and
>>>> > tools around in the Java world for doing string manipulations and such,
>>>> > they don't need to be built into a template language. Taking that one
>>>> step
>>>> > further, I'd actually prefer that the expression syntax in the template
>>>> > language makes it easy to call an existing API vs trying to make
>>>> everything
>>>> > built into the template language.
>>>> >
>>>> > For example in addition to using FTL I also use Groovy heavily and Groovy
>>>> > already has a great API including standard Java API augmentation for all
>>>> > sorts of things. Yes, FTL could use the Groovy String and other expanded
>>>> > objects as a reference for built in design and have a good set of
>>>> commonly
>>>> > needed things, but why not just defer to Groovy for those that use it or
>>>> > whatever other API/library anyone might prefer.
>>>> >
>>>> > I know this is a VERY different direction from FM2. What I find most
>>>> useful
>>>> > in FTL is features like macros and the extensive directives and other
>>>> > things that make a template language flexible and powerful. The built-ins
>>>> > are handy sometimes, but often don't do what I need anyway so other than
>>>> > things like ?has_content I don't use them a lot.
>>>>
>>>> An important (and as far as I'm concerned, the main) reason for many
>>>> built-ins is/was that frameworks and other products drop in FreeMarker
>>>> for templates and then often don't give much attention to give tools
>>>> to the template authors. Then the poor people who actually has to
>>>> create templates come and ask, "I must do X... how can I do that in
>>>> FreeMaker?". The less often we have to say "Please pester the
>>>> developers (who are often from another company, or are on another
>>>> sub-project by now => you have near 0 chance) to expose such
>>>> functionality to the templates", the better. Instead we can say "Like
>>>> foo?x", and furthermore, ?x is documented, and people can actually
>>>> find its documentation, even years later. Surely if you have some
>>>> small company with the sharp people, this is not a problem for you.
>>>> Actually, then you very often don't want the one-size-fits-all
>>>> solutions anyway.
>>>>
>>>> > I've mentioned this before but what would be a killer feature for FTL is
>>>> to
>>>> > support pluggable expression evaluators. The FTL expression syntax isn't
>>>> > awful
>>>>
>>>> (Yes it is... but that's besides the point until I have fixed it in
>>>> FM3.)
>>>>
>>>> > but is not nearly as convenient as Groovy, especially when you are
>>>> > using it extensively elsewhere.
>>>>
>>>> The theoretical problem is of course that both languages (the "outer"
>>>> template language and the language from where you rip off the
>>>> expression language from) have its own ideas, and so there will be
>>>> mismatches. The expression language in a language and other parts of
>>>> the language aren't separate in general anyway. Which leads to the
>>>> question that if you are using Groovy expressions, why not also use
>>>> for loops and whatnot from Groovy? (This is fairly common way of
>>>> creating template language, just taking an existing language, and add
>>>> some convenience syntax to embed it into the static text.) So then we
>>>> don't need our own #list, #if, etc. implementation either. But then,
>>>> the original idea was that a template language is a domain specific
>>>> language, and as such can have more specialized, say, loop constructs
>>>> for that domain. (I'm not saying that FM2 did great there, but there
>>>> was a lack of determination there as well.) If that's not the case
>>>> though...
>>>>
>>>> Anyway, when and if FM3 will be more mature, we can see if how doable
>>>> this is in reality. It's also a question if we are talking about a
>>>> syntax, that in 99% looks like Groovy, or we are talking about actual
>>>> Groovy. In the last case TemplateModel-s and the particular way null
>>>> handling will work will be an obstacle for sure. Not to mention
>>>> TemplateMarkupOutputModel-s, that AFAIK don't even have commonly used
>>>> equivalent (like a class in the standard Java API).
>>>>
>>>> > If the default FTL expressions were not used you couldn't use
>>>> > built-ins and instead would just have expressions that are expected
>>>> > to evaluate to a boolean, string, or arbitrary object and can be
>>>> > whatever.
>>>>
>>>> At least in FM3 built-ins will be just functions you have defined in
>>>> the "core" FTL namespace (whose content is specified in template
>>>> language Dialect you use) that's "statically linked" to the templates
>>>> (allowing some parse-time checks and somewhat faster calls). So
>>>> `foo?bar(1)` is basically just a shorthand syntax for `<#import core =
>>>> 'freemarker://dialect-or-something'>` and `core.bar(foo, 1)`. There's
>>>> a standard/default Dialect of course, which has the big advantage that
>>>> it's documented in the official Manual, is used in SO answers, etc.
>>>>
>>>> > How important is this? Important enough that I've
>>>> > considered using a different template language. If FTL wasn't so
>>>> > darn convenience for XML transformation using macros I'd probably be
>>>> > using Groovy GString templates instead, and just for the far better
>>>> > syntax in Groovy.
>>>> >
>>>> > This also goes back to the object wrapping discussion which is another
>>>> can
>>>> > of worms...
>>>>
>>>> Yeah, it's one of the things that I really wanted to get rid of, but
>>>> it's a very disrupting transition to do. I don't mean for most of the
>>>> users, but for the source code. On the brighter side, what really
>>>> happens when you do that transition is that instead of something like:
>>>>
>>>>   if (value is TemplateMapLikeModel) {
>>>>       ... ((TemplateMapLikeModel) value).get(k) ...
>>>>   }
>>>>
>>>> you, in first approach, rewrite it to something like:
>>>>
>>>>   if (mop.isMapLike(value)) {
>>>>       ... mop.get(value, k) ...
>>>>   }
>>>>
>>>> Of course that would be a silly MOP API design, so this below is more
>>>> likely, but now it's visible that one comes from the other trivially:
>>>>
>>>>
>>>>   MapLike m = mop.tryGetAsMapLike(value);
>>>>   if (m != null) {
>>>>       ... m.get(k) ...
>>>>   }
>>>>
>>>> Hence when we have our TemplateModel-s cleaned up, it's easier to
>>>> migrate to the MOP approach. The "type system" (just a mental model
>>>> really) depicted be the TemplateModel still exists, you just use a MOP
>>>> to implement it.
>>>>
>>>> > I don't mean to derail this discussion, but especially these days in a
>>>> > world of a huge number of great Java libraries out there it is often
>>>> better
>>>> > to not try to do everything and instead focus on core strengths that you
>>>> > can't get anywhere else, and make it easy for people to use other open
>>>> > source projects for their core strengths. This could even go as far as
>>>> > FreeMarker builds that have dependencies on certain commonly used open
>>>> > source projects, there are many even in the ASF that do various things
>>>> that
>>>> > FTL built-ins do but with far more options and well known behavior for
>>>> the
>>>> > many who use them (like various Commons projects).
>>>>
>>>> It's really the duty of the Dialects feature of FM3. You can put
>>>> together your "toolbox" from whatever you want, like call Apache
>>>> Commons. I mean it's unrelated to question of expression languages, as
>>>> far as I see. (Like, you can call plain Java methods from FM2 as
>>>> well.)
>>>>
>>>> If we want to use Apache Commons etc. in implementing the functions of
>>>> the standard dialect... I *guess* we won't benefit from it much at
>>>> this point, but I need to see it case-by-case to tell, like, the FM
>>>> implementation of function x sucks, while Apache Commons StringUtils.x
>>>> is great, etc. In any case, users don't care, as far as they can call
>>>> ?x and it does its job. My point in this thread is that we should be
>>>> able to tell users to "just write ?x" (see earlier), instead of add
>>>> your own solution somehow. (Also, if ?x works, they prefer if we have
>>>> less dependencies. Especially if we want to support Andorid.)
>>>>
>>>> > Would supporting different expression languages cause huge problems for
>>>> IDE
>>>> > support? Possibly, but IDEs these days have lots of support for language
>>>> > injection and layers that are meant for just this sort of thing.
>>>>
>>>> At least looking from big enough distance it's easy. :) Note that so
>>>> far we fail to deliver an editor for Eclipse that integrates the
>>>> existing HTML editor functionality with FTL... :-> Please everyone go
>>>> and help out Angelo Zerr so that this will change:
>>>>
>>>> https://github.com/angelozerr/lsp4e-freemarker
>>>> https://github.com/angelozerr/freemarker-languageserver
>>>>
>>>> Anyway, IDE support won't be the obstacle. At worse only the default
>>>> expression syntax will have reasonable IDE support.
>>>>
>>>> > In theory FreeMarker itself could be a heck of a lot smaller and
>>>> > simpler and at the same time be far more flexible and powerful. IMO
>>>> > that could be a game changer in the world of Java-based template
>>>> > languages. Some of this would be more work than not supporting this
>>>> > sort of flexibility (ie support Groovy syntax as an option, ie
>>>> > without requiring the much larger Groovy jar files as dependencies
>>>> > by having a default built-in expression syntax), but by simplifying
>>>> > other things it would save a heck of a lot of work (including design
>>>> > work).
>>>>
>>>> So, I'm not sure I follow what exactly are you proposing. To use
>>>> vanilla Groovy (the actual implementation) for expressions? How does
>>>> that blend with the directives (which mostly also have similar pairs
>>>> in Groovy), and other (planned) FM features? (And... why's the result
>>>> called FreeMarker at all? ;) )
>>>>
>>>> > -David
>>>> >
>>>> >
>>>> >
>>>> > On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <
>>>> slidingfilaments@gmail.com
>>>> >> wrote:
>>>> >
>>>> >> I'm not a subject matter expert here, but I always thought sensible
>>>> >> defaults with the ability to override using configuration is a wise
>>>> >> move because you appeal to many people out of the box and then allow
>>>> >> minority to also not feel locked out when they need to override. Also,
>>>> >> as a general rule, I found from the projects we worked on that
>>>> >> internationalization is always a challenge, but you can also resolve
>>>> >> that with the same solution (sensible defaults with configuration
>>>> >> overrides); then you simply let the community adopt and improve the
>>>> >> code for non-English parts.
>>>> >>
>>>> >> Of course then the challenge becomes, what's a sensible default? I
>>>> >> wish I had a dollar every time I asked that! I'd favor simplicity
>>>> >> where possible. In your example above maybe "No space f..." is the
>>>> >> simplest, and then people can get fancy if they want to?
>>>> >>
>>>> >> My 2 cents, sorry for the noise :)
>>>> >>
>>>> >> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <dd...@apache.org>
>>>> >> wrote:
>>>> >> > There's this old frequently requested built-in,
>>>> >> > maybeTooLongText?truncate(n). Like "No space for such a long
>>>> >> > title"?truncate(13) would give "No space...". The problem with these
>>>> >> > is that there's no single algorithm for this that fits the need of
>>>> >> > everyone. Like maybe the result should be "No space f...", or "No
>>>> >> > space [...]", etc. This can even change depending on the current value
>>>> >> > of the `locale` FreeMarker setting.
>>>> >> >
>>>> >> > Now there's another similar wish, for "?title_case", which has the
>>>> >> > same problem, plus it's much more locale dependent. See:
>>>> >> >
>>>> >> > https://issues.apache.org/jira/browse/FREEMARKER-97
>>>> >> >
>>>> >> > We do provide locale dependent functions, like number and date
>>>> >> > formatting. This is pretty much an FM tradition, to treat non-English
>>>> >> > users as equals (well, except that the keywords aren't localizable).
>>>> >> > However, we push those tasks to the Java platform. But it doesn't
>>>> >> > support title case (as far as I know). Besides there can be different
>>>> >> > preferences even inside the same country, just as with ?truncate.
>>>> >> >
>>>> >> > So, it's acceptable to chicken out ;) saying that none of the above
>>>> >> > has a widely accepted default, so the correct solution is letting the
>>>> >> > user solve this. But I think in practice what happens is that people
>>>> >> > need the functionality (especially ?truncate), and then has to come up
>>>> >> > with some ad-hoc solution on the spot, which will be almost always
>>>> >> > worse than the default we could provide. So we might as well just
>>>> >> > provide some decent default, sparing time for lot of users. Then, then
>>>> >> > if they are particular about the algorithm (which is certainly rare
>>>> >> > nowadays), let them plug their own via the Configurable API. So there
>>>> >> > would be something like
>>>> >> > cfg.setTruncateImplementation(localToImplementationMap), where the
>>>> >> > parameter maps locale-s to Java functions and defines a fallback
>>>> >> > function. (I haven't worked out the API details, as you can see.)
>>>> >> >
>>>> >> > What do you think?
>>>> >> >
>>>> >> > --
>>>> >> > Thanks,
>>>> >> >  Daniel Dekany
>>>> >> >
>>>> >>
>>>>
>>>> --
>>>> Thanks,
>>>>  Daniel Dekany
>>>>
>>>>
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>

-- 
Thanks,
 Daniel Dekany

Re: The problem of "too human" text transformations

[Woonsan Ko <wo...@apache.org>]

On Mon, Jun 4, 2018 at 5:04 AM, Daniel Dekany <dd...@apache.org> wrote:
> Sunday, June 3, 2018, 10:13:04 PM, David E Jones wrote:
>
>> You make good points Daniel. At a high level it reminds me of a discussion
>> with a new client a couple months ago on the topic of managing templates
>> for ecommerce sites. They want to use the Liquid template engine partly
>> because they are doing custom ecommerce for some customers and Shopify
>> (which uses Liquid, where Liquid came from) for others, and also because
>> Liquid is more constrained and easier for web developers/designers to use.
>>
>> There is a case for template languages that are less developer oriented and
>> more designer oriented, or maybe better put because visual web designers
>> have no business mucking around with templates server side developer vs
>> web/browser developer (even if this is stuff that generally runs on a
>> server anyway).
>>
>> Right now, ie FM2, FreeMarker strikes a good balance between these (IMO
>> anyway), and the built-ins are an important part of that. Maybe they don't
>> do every little thing everyone in the world might want but they cover a lot
>> of basics and common needs.
>>
>> At a lower level you are correct about template parsing and supporting
>> arbitrary languages for inline expressions. The template parser still needs
>> to be able to figure out where the expression starts and stops. This might
>> not be too bad for C-like languages which current FM expressions are (more
>> or less, mostly the same demarcation characters and all), but is still a
>> challenge and could require something like surrounding expressions with
>> parenthesis (along with nested parenthesis evaluation that FM already does)
>> and that's a little bit ugly.
>
> I wasn't even worried about that yet, but the semantical differences.
> The expression and non-expression language (if they are separate at
> all) build on a common mental model, so you can't, in general, cut out
> the expression language from one language, and put it into another. So
> I can only look at this on a case-by-case basis.
>
> As of the syntactical problems, at least if I write the expression
> parser, I can just exit expression mode if the next token would lead
> to an error, and let the "outer language" process the next character.
> Like you have `a + b n=2`, then `n` would be an error, and the outer
> parser will pick it up and recognize that as a parameter assignment.
> What's tricky is making JavaCC to do this... it doesn't support
> dynamic composing by design. ANTLR, I don't know (nor if it's fast
> enough for use cases like a lot of ?eval/?interpret). For a hand
> written parser, it's certainly not a problem. Though switching to hand
> written is a questionable step, even if I personally find it tempting.
>
>> Anyway, sorry to derail the discussion.
>>
>> On the truncate built-in an optional String type second argument makes
>> sense as a suffix to use if the string is truncated, ie that seems like an
>> easy thing to accommodate without reducing flexibility or having to write
>> that logic to do that inline in the template.
>
> There's also the question of where to cut the text. There's the
> maximum length given as parameter (actually, you subtract the length
> of the cut marker suffix), however if with applying that naively you
> end up with white-space at the end, you probably want to trim that.
> Also certainly if it ends with a ellipsis, dot or a sequence of dots,
> and maybe even if with some other punctuation. There's also the
> typical preference, over a certain maximum length, where you only cut
> at the end of words (unless you have some extremely long word).

Commons-text's WordUtils#abbreviate(str, lower, upper, appendToEnd)
finds a space from the lower index to find where to cut.
It doesn't care other white spaces nor ending dot(s), but one of
existing simple algorithms at the moment. ;-)

>
>> On title casing it might be best to defer to what Java does (ie
>> Character.toUpperCase() or .toTitleCase()), or were there issues with that?
>
> They don't convert text to title case, only a single character. In the
> most basic form of title case conversion of text, you convert the
> first letter of each word to title case. You may already have some
> tricky parts there with finding word boundaries. Then, in many
> languages, like in English, you aren't supported to convert words like
> "and", "of", etc. So it's obviously locale dependent.

WordUtils#capitalizeFully() does. It doesn't recognize "and", "of", etc. though.

But as you said, if it can be customized through Configuration for
locale specific use cases, I'd say it's perfect.

Regards,

Woonsan

>
>> -David
>>
>>
>> On Thu, May 31, 2018 at 2:03 AM, Daniel Dekany <dd...@apache.org> wrote:
>>
>>> Thursday, May 31, 2018, 1:16:26 AM, David E Jones wrote:
>>>
>>> > My general opinion on this sort of thing is there are lots of libraries
>>> and
>>> > tools around in the Java world for doing string manipulations and such,
>>> > they don't need to be built into a template language. Taking that one
>>> step
>>> > further, I'd actually prefer that the expression syntax in the template
>>> > language makes it easy to call an existing API vs trying to make
>>> everything
>>> > built into the template language.
>>> >
>>> > For example in addition to using FTL I also use Groovy heavily and Groovy
>>> > already has a great API including standard Java API augmentation for all
>>> > sorts of things. Yes, FTL could use the Groovy String and other expanded
>>> > objects as a reference for built in design and have a good set of
>>> commonly
>>> > needed things, but why not just defer to Groovy for those that use it or
>>> > whatever other API/library anyone might prefer.
>>> >
>>> > I know this is a VERY different direction from FM2. What I find most
>>> useful
>>> > in FTL is features like macros and the extensive directives and other
>>> > things that make a template language flexible and powerful. The built-ins
>>> > are handy sometimes, but often don't do what I need anyway so other than
>>> > things like ?has_content I don't use them a lot.
>>>
>>> An important (and as far as I'm concerned, the main) reason for many
>>> built-ins is/was that frameworks and other products drop in FreeMarker
>>> for templates and then often don't give much attention to give tools
>>> to the template authors. Then the poor people who actually has to
>>> create templates come and ask, "I must do X... how can I do that in
>>> FreeMaker?". The less often we have to say "Please pester the
>>> developers (who are often from another company, or are on another
>>> sub-project by now => you have near 0 chance) to expose such
>>> functionality to the templates", the better. Instead we can say "Like
>>> foo?x", and furthermore, ?x is documented, and people can actually
>>> find its documentation, even years later. Surely if you have some
>>> small company with the sharp people, this is not a problem for you.
>>> Actually, then you very often don't want the one-size-fits-all
>>> solutions anyway.
>>>
>>> > I've mentioned this before but what would be a killer feature for FTL is
>>> to
>>> > support pluggable expression evaluators. The FTL expression syntax isn't
>>> > awful
>>>
>>> (Yes it is... but that's besides the point until I have fixed it in
>>> FM3.)
>>>
>>> > but is not nearly as convenient as Groovy, especially when you are
>>> > using it extensively elsewhere.
>>>
>>> The theoretical problem is of course that both languages (the "outer"
>>> template language and the language from where you rip off the
>>> expression language from) have its own ideas, and so there will be
>>> mismatches. The expression language in a language and other parts of
>>> the language aren't separate in general anyway. Which leads to the
>>> question that if you are using Groovy expressions, why not also use
>>> for loops and whatnot from Groovy? (This is fairly common way of
>>> creating template language, just taking an existing language, and add
>>> some convenience syntax to embed it into the static text.) So then we
>>> don't need our own #list, #if, etc. implementation either. But then,
>>> the original idea was that a template language is a domain specific
>>> language, and as such can have more specialized, say, loop constructs
>>> for that domain. (I'm not saying that FM2 did great there, but there
>>> was a lack of determination there as well.) If that's not the case
>>> though...
>>>
>>> Anyway, when and if FM3 will be more mature, we can see if how doable
>>> this is in reality. It's also a question if we are talking about a
>>> syntax, that in 99% looks like Groovy, or we are talking about actual
>>> Groovy. In the last case TemplateModel-s and the particular way null
>>> handling will work will be an obstacle for sure. Not to mention
>>> TemplateMarkupOutputModel-s, that AFAIK don't even have commonly used
>>> equivalent (like a class in the standard Java API).
>>>
>>> > If the default FTL expressions were not used you couldn't use
>>> > built-ins and instead would just have expressions that are expected
>>> > to evaluate to a boolean, string, or arbitrary object and can be
>>> > whatever.
>>>
>>> At least in FM3 built-ins will be just functions you have defined in
>>> the "core" FTL namespace (whose content is specified in template
>>> language Dialect you use) that's "statically linked" to the templates
>>> (allowing some parse-time checks and somewhat faster calls). So
>>> `foo?bar(1)` is basically just a shorthand syntax for `<#import core =
>>> 'freemarker://dialect-or-something'>` and `core.bar(foo, 1)`. There's
>>> a standard/default Dialect of course, which has the big advantage that
>>> it's documented in the official Manual, is used in SO answers, etc.
>>>
>>> > How important is this? Important enough that I've
>>> > considered using a different template language. If FTL wasn't so
>>> > darn convenience for XML transformation using macros I'd probably be
>>> > using Groovy GString templates instead, and just for the far better
>>> > syntax in Groovy.
>>> >
>>> > This also goes back to the object wrapping discussion which is another
>>> can
>>> > of worms...
>>>
>>> Yeah, it's one of the things that I really wanted to get rid of, but
>>> it's a very disrupting transition to do. I don't mean for most of the
>>> users, but for the source code. On the brighter side, what really
>>> happens when you do that transition is that instead of something like:
>>>
>>>   if (value is TemplateMapLikeModel) {
>>>       ... ((TemplateMapLikeModel) value).get(k) ...
>>>   }
>>>
>>> you, in first approach, rewrite it to something like:
>>>
>>>   if (mop.isMapLike(value)) {
>>>       ... mop.get(value, k) ...
>>>   }
>>>
>>> Of course that would be a silly MOP API design, so this below is more
>>> likely, but now it's visible that one comes from the other trivially:
>>>
>>>
>>>   MapLike m = mop.tryGetAsMapLike(value);
>>>   if (m != null) {
>>>       ... m.get(k) ...
>>>   }
>>>
>>> Hence when we have our TemplateModel-s cleaned up, it's easier to
>>> migrate to the MOP approach. The "type system" (just a mental model
>>> really) depicted be the TemplateModel still exists, you just use a MOP
>>> to implement it.
>>>
>>> > I don't mean to derail this discussion, but especially these days in a
>>> > world of a huge number of great Java libraries out there it is often
>>> better
>>> > to not try to do everything and instead focus on core strengths that you
>>> > can't get anywhere else, and make it easy for people to use other open
>>> > source projects for their core strengths. This could even go as far as
>>> > FreeMarker builds that have dependencies on certain commonly used open
>>> > source projects, there are many even in the ASF that do various things
>>> that
>>> > FTL built-ins do but with far more options and well known behavior for
>>> the
>>> > many who use them (like various Commons projects).
>>>
>>> It's really the duty of the Dialects feature of FM3. You can put
>>> together your "toolbox" from whatever you want, like call Apache
>>> Commons. I mean it's unrelated to question of expression languages, as
>>> far as I see. (Like, you can call plain Java methods from FM2 as
>>> well.)
>>>
>>> If we want to use Apache Commons etc. in implementing the functions of
>>> the standard dialect... I *guess* we won't benefit from it much at
>>> this point, but I need to see it case-by-case to tell, like, the FM
>>> implementation of function x sucks, while Apache Commons StringUtils.x
>>> is great, etc. In any case, users don't care, as far as they can call
>>> ?x and it does its job. My point in this thread is that we should be
>>> able to tell users to "just write ?x" (see earlier), instead of add
>>> your own solution somehow. (Also, if ?x works, they prefer if we have
>>> less dependencies. Especially if we want to support Andorid.)
>>>
>>> > Would supporting different expression languages cause huge problems for
>>> IDE
>>> > support? Possibly, but IDEs these days have lots of support for language
>>> > injection and layers that are meant for just this sort of thing.
>>>
>>> At least looking from big enough distance it's easy. :) Note that so
>>> far we fail to deliver an editor for Eclipse that integrates the
>>> existing HTML editor functionality with FTL... :-> Please everyone go
>>> and help out Angelo Zerr so that this will change:
>>>
>>> https://github.com/angelozerr/lsp4e-freemarker
>>> https://github.com/angelozerr/freemarker-languageserver
>>>
>>> Anyway, IDE support won't be the obstacle. At worse only the default
>>> expression syntax will have reasonable IDE support.
>>>
>>> > In theory FreeMarker itself could be a heck of a lot smaller and
>>> > simpler and at the same time be far more flexible and powerful. IMO
>>> > that could be a game changer in the world of Java-based template
>>> > languages. Some of this would be more work than not supporting this
>>> > sort of flexibility (ie support Groovy syntax as an option, ie
>>> > without requiring the much larger Groovy jar files as dependencies
>>> > by having a default built-in expression syntax), but by simplifying
>>> > other things it would save a heck of a lot of work (including design
>>> > work).
>>>
>>> So, I'm not sure I follow what exactly are you proposing. To use
>>> vanilla Groovy (the actual implementation) for expressions? How does
>>> that blend with the directives (which mostly also have similar pairs
>>> in Groovy), and other (planned) FM features? (And... why's the result
>>> called FreeMarker at all? ;) )
>>>
>>> > -David
>>> >
>>> >
>>> >
>>> > On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <
>>> slidingfilaments@gmail.com
>>> >> wrote:
>>> >
>>> >> I'm not a subject matter expert here, but I always thought sensible
>>> >> defaults with the ability to override using configuration is a wise
>>> >> move because you appeal to many people out of the box and then allow
>>> >> minority to also not feel locked out when they need to override. Also,
>>> >> as a general rule, I found from the projects we worked on that
>>> >> internationalization is always a challenge, but you can also resolve
>>> >> that with the same solution (sensible defaults with configuration
>>> >> overrides); then you simply let the community adopt and improve the
>>> >> code for non-English parts.
>>> >>
>>> >> Of course then the challenge becomes, what's a sensible default? I
>>> >> wish I had a dollar every time I asked that! I'd favor simplicity
>>> >> where possible. In your example above maybe "No space f..." is the
>>> >> simplest, and then people can get fancy if they want to?
>>> >>
>>> >> My 2 cents, sorry for the noise :)
>>> >>
>>> >> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <dd...@apache.org>
>>> >> wrote:
>>> >> > There's this old frequently requested built-in,
>>> >> > maybeTooLongText?truncate(n). Like "No space for such a long
>>> >> > title"?truncate(13) would give "No space...". The problem with these
>>> >> > is that there's no single algorithm for this that fits the need of
>>> >> > everyone. Like maybe the result should be "No space f...", or "No
>>> >> > space [...]", etc. This can even change depending on the current value
>>> >> > of the `locale` FreeMarker setting.
>>> >> >
>>> >> > Now there's another similar wish, for "?title_case", which has the
>>> >> > same problem, plus it's much more locale dependent. See:
>>> >> >
>>> >> > https://issues.apache.org/jira/browse/FREEMARKER-97
>>> >> >
>>> >> > We do provide locale dependent functions, like number and date
>>> >> > formatting. This is pretty much an FM tradition, to treat non-English
>>> >> > users as equals (well, except that the keywords aren't localizable).
>>> >> > However, we push those tasks to the Java platform. But it doesn't
>>> >> > support title case (as far as I know). Besides there can be different
>>> >> > preferences even inside the same country, just as with ?truncate.
>>> >> >
>>> >> > So, it's acceptable to chicken out ;) saying that none of the above
>>> >> > has a widely accepted default, so the correct solution is letting the
>>> >> > user solve this. But I think in practice what happens is that people
>>> >> > need the functionality (especially ?truncate), and then has to come up
>>> >> > with some ad-hoc solution on the spot, which will be almost always
>>> >> > worse than the default we could provide. So we might as well just
>>> >> > provide some decent default, sparing time for lot of users. Then, then
>>> >> > if they are particular about the algorithm (which is certainly rare
>>> >> > nowadays), let them plug their own via the Configurable API. So there
>>> >> > would be something like
>>> >> > cfg.setTruncateImplementation(localToImplementationMap), where the
>>> >> > parameter maps locale-s to Java functions and defines a fallback
>>> >> > function. (I haven't worked out the API details, as you can see.)
>>> >> >
>>> >> > What do you think?
>>> >> >
>>> >> > --
>>> >> > Thanks,
>>> >> >  Daniel Dekany
>>> >> >
>>> >>
>>>
>>> --
>>> Thanks,
>>>  Daniel Dekany
>>>
>>>
>
> --
> Thanks,
>  Daniel Dekany
>

Re: The problem of "too human" text transformations

[Daniel Dekany <dd...@apache.org>]

Sunday, June 3, 2018, 10:13:04 PM, David E Jones wrote:

> You make good points Daniel. At a high level it reminds me of a discussion
> with a new client a couple months ago on the topic of managing templates
> for ecommerce sites. They want to use the Liquid template engine partly
> because they are doing custom ecommerce for some customers and Shopify
> (which uses Liquid, where Liquid came from) for others, and also because
> Liquid is more constrained and easier for web developers/designers to use.
>
> There is a case for template languages that are less developer oriented and
> more designer oriented, or maybe better put because visual web designers
> have no business mucking around with templates server side developer vs
> web/browser developer (even if this is stuff that generally runs on a
> server anyway).
>
> Right now, ie FM2, FreeMarker strikes a good balance between these (IMO
> anyway), and the built-ins are an important part of that. Maybe they don't
> do every little thing everyone in the world might want but they cover a lot
> of basics and common needs.
>
> At a lower level you are correct about template parsing and supporting
> arbitrary languages for inline expressions. The template parser still needs
> to be able to figure out where the expression starts and stops. This might
> not be too bad for C-like languages which current FM expressions are (more
> or less, mostly the same demarcation characters and all), but is still a
> challenge and could require something like surrounding expressions with
> parenthesis (along with nested parenthesis evaluation that FM already does)
> and that's a little bit ugly.

I wasn't even worried about that yet, but the semantical differences.
The expression and non-expression language (if they are separate at
all) build on a common mental model, so you can't, in general, cut out
the expression language from one language, and put it into another. So
I can only look at this on a case-by-case basis.

As of the syntactical problems, at least if I write the expression
parser, I can just exit expression mode if the next token would lead
to an error, and let the "outer language" process the next character.
Like you have `a + b n=2`, then `n` would be an error, and the outer
parser will pick it up and recognize that as a parameter assignment.
What's tricky is making JavaCC to do this... it doesn't support
dynamic composing by design. ANTLR, I don't know (nor if it's fast
enough for use cases like a lot of ?eval/?interpret). For a hand
written parser, it's certainly not a problem. Though switching to hand
written is a questionable step, even if I personally find it tempting.

> Anyway, sorry to derail the discussion.
>
> On the truncate built-in an optional String type second argument makes
> sense as a suffix to use if the string is truncated, ie that seems like an
> easy thing to accommodate without reducing flexibility or having to write
> that logic to do that inline in the template.

There's also the question of where to cut the text. There's the
maximum length given as parameter (actually, you subtract the length
of the cut marker suffix), however if with applying that naively you
end up with white-space at the end, you probably want to trim that.
Also certainly if it ends with a ellipsis, dot or a sequence of dots,
and maybe even if with some other punctuation. There's also the
typical preference, over a certain maximum length, where you only cut
at the end of words (unless you have some extremely long word).

> On title casing it might be best to defer to what Java does (ie
> Character.toUpperCase() or .toTitleCase()), or were there issues with that?

They don't convert text to title case, only a single character. In the
most basic form of title case conversion of text, you convert the
first letter of each word to title case. You may already have some
tricky parts there with finding word boundaries. Then, in many
languages, like in English, you aren't supported to convert words like
"and", "of", etc. So it's obviously locale dependent.

> -David
>
>
> On Thu, May 31, 2018 at 2:03 AM, Daniel Dekany <dd...@apache.org> wrote:
>
>> Thursday, May 31, 2018, 1:16:26 AM, David E Jones wrote:
>>
>> > My general opinion on this sort of thing is there are lots of libraries
>> and
>> > tools around in the Java world for doing string manipulations and such,
>> > they don't need to be built into a template language. Taking that one
>> step
>> > further, I'd actually prefer that the expression syntax in the template
>> > language makes it easy to call an existing API vs trying to make
>> everything
>> > built into the template language.
>> >
>> > For example in addition to using FTL I also use Groovy heavily and Groovy
>> > already has a great API including standard Java API augmentation for all
>> > sorts of things. Yes, FTL could use the Groovy String and other expanded
>> > objects as a reference for built in design and have a good set of
>> commonly
>> > needed things, but why not just defer to Groovy for those that use it or
>> > whatever other API/library anyone might prefer.
>> >
>> > I know this is a VERY different direction from FM2. What I find most
>> useful
>> > in FTL is features like macros and the extensive directives and other
>> > things that make a template language flexible and powerful. The built-ins
>> > are handy sometimes, but often don't do what I need anyway so other than
>> > things like ?has_content I don't use them a lot.
>>
>> An important (and as far as I'm concerned, the main) reason for many
>> built-ins is/was that frameworks and other products drop in FreeMarker
>> for templates and then often don't give much attention to give tools
>> to the template authors. Then the poor people who actually has to
>> create templates come and ask, "I must do X... how can I do that in
>> FreeMaker?". The less often we have to say "Please pester the
>> developers (who are often from another company, or are on another
>> sub-project by now => you have near 0 chance) to expose such
>> functionality to the templates", the better. Instead we can say "Like
>> foo?x", and furthermore, ?x is documented, and people can actually
>> find its documentation, even years later. Surely if you have some
>> small company with the sharp people, this is not a problem for you.
>> Actually, then you very often don't want the one-size-fits-all
>> solutions anyway.
>>
>> > I've mentioned this before but what would be a killer feature for FTL is
>> to
>> > support pluggable expression evaluators. The FTL expression syntax isn't
>> > awful
>>
>> (Yes it is... but that's besides the point until I have fixed it in
>> FM3.)
>>
>> > but is not nearly as convenient as Groovy, especially when you are
>> > using it extensively elsewhere.
>>
>> The theoretical problem is of course that both languages (the "outer"
>> template language and the language from where you rip off the
>> expression language from) have its own ideas, and so there will be
>> mismatches. The expression language in a language and other parts of
>> the language aren't separate in general anyway. Which leads to the
>> question that if you are using Groovy expressions, why not also use
>> for loops and whatnot from Groovy? (This is fairly common way of
>> creating template language, just taking an existing language, and add
>> some convenience syntax to embed it into the static text.) So then we
>> don't need our own #list, #if, etc. implementation either. But then,
>> the original idea was that a template language is a domain specific
>> language, and as such can have more specialized, say, loop constructs
>> for that domain. (I'm not saying that FM2 did great there, but there
>> was a lack of determination there as well.) If that's not the case
>> though...
>>
>> Anyway, when and if FM3 will be more mature, we can see if how doable
>> this is in reality. It's also a question if we are talking about a
>> syntax, that in 99% looks like Groovy, or we are talking about actual
>> Groovy. In the last case TemplateModel-s and the particular way null
>> handling will work will be an obstacle for sure. Not to mention
>> TemplateMarkupOutputModel-s, that AFAIK don't even have commonly used
>> equivalent (like a class in the standard Java API).
>>
>> > If the default FTL expressions were not used you couldn't use
>> > built-ins and instead would just have expressions that are expected
>> > to evaluate to a boolean, string, or arbitrary object and can be
>> > whatever.
>>
>> At least in FM3 built-ins will be just functions you have defined in
>> the "core" FTL namespace (whose content is specified in template
>> language Dialect you use) that's "statically linked" to the templates
>> (allowing some parse-time checks and somewhat faster calls). So
>> `foo?bar(1)` is basically just a shorthand syntax for `<#import core =
>> 'freemarker://dialect-or-something'>` and `core.bar(foo, 1)`. There's
>> a standard/default Dialect of course, which has the big advantage that
>> it's documented in the official Manual, is used in SO answers, etc.
>>
>> > How important is this? Important enough that I've
>> > considered using a different template language. If FTL wasn't so
>> > darn convenience for XML transformation using macros I'd probably be
>> > using Groovy GString templates instead, and just for the far better
>> > syntax in Groovy.
>> >
>> > This also goes back to the object wrapping discussion which is another
>> can
>> > of worms...
>>
>> Yeah, it's one of the things that I really wanted to get rid of, but
>> it's a very disrupting transition to do. I don't mean for most of the
>> users, but for the source code. On the brighter side, what really
>> happens when you do that transition is that instead of something like:
>>
>>   if (value is TemplateMapLikeModel) {
>>       ... ((TemplateMapLikeModel) value).get(k) ...
>>   }
>>
>> you, in first approach, rewrite it to something like:
>>
>>   if (mop.isMapLike(value)) {
>>       ... mop.get(value, k) ...
>>   }
>>
>> Of course that would be a silly MOP API design, so this below is more
>> likely, but now it's visible that one comes from the other trivially:
>>
>>
>>   MapLike m = mop.tryGetAsMapLike(value);
>>   if (m != null) {
>>       ... m.get(k) ...
>>   }
>>
>> Hence when we have our TemplateModel-s cleaned up, it's easier to
>> migrate to the MOP approach. The "type system" (just a mental model
>> really) depicted be the TemplateModel still exists, you just use a MOP
>> to implement it.
>>
>> > I don't mean to derail this discussion, but especially these days in a
>> > world of a huge number of great Java libraries out there it is often
>> better
>> > to not try to do everything and instead focus on core strengths that you
>> > can't get anywhere else, and make it easy for people to use other open
>> > source projects for their core strengths. This could even go as far as
>> > FreeMarker builds that have dependencies on certain commonly used open
>> > source projects, there are many even in the ASF that do various things
>> that
>> > FTL built-ins do but with far more options and well known behavior for
>> the
>> > many who use them (like various Commons projects).
>>
>> It's really the duty of the Dialects feature of FM3. You can put
>> together your "toolbox" from whatever you want, like call Apache
>> Commons. I mean it's unrelated to question of expression languages, as
>> far as I see. (Like, you can call plain Java methods from FM2 as
>> well.)
>>
>> If we want to use Apache Commons etc. in implementing the functions of
>> the standard dialect... I *guess* we won't benefit from it much at
>> this point, but I need to see it case-by-case to tell, like, the FM
>> implementation of function x sucks, while Apache Commons StringUtils.x
>> is great, etc. In any case, users don't care, as far as they can call
>> ?x and it does its job. My point in this thread is that we should be
>> able to tell users to "just write ?x" (see earlier), instead of add
>> your own solution somehow. (Also, if ?x works, they prefer if we have
>> less dependencies. Especially if we want to support Andorid.)
>>
>> > Would supporting different expression languages cause huge problems for
>> IDE
>> > support? Possibly, but IDEs these days have lots of support for language
>> > injection and layers that are meant for just this sort of thing.
>>
>> At least looking from big enough distance it's easy. :) Note that so
>> far we fail to deliver an editor for Eclipse that integrates the
>> existing HTML editor functionality with FTL... :-> Please everyone go
>> and help out Angelo Zerr so that this will change:
>>
>> https://github.com/angelozerr/lsp4e-freemarker
>> https://github.com/angelozerr/freemarker-languageserver
>>
>> Anyway, IDE support won't be the obstacle. At worse only the default
>> expression syntax will have reasonable IDE support.
>>
>> > In theory FreeMarker itself could be a heck of a lot smaller and
>> > simpler and at the same time be far more flexible and powerful. IMO
>> > that could be a game changer in the world of Java-based template
>> > languages. Some of this would be more work than not supporting this
>> > sort of flexibility (ie support Groovy syntax as an option, ie
>> > without requiring the much larger Groovy jar files as dependencies
>> > by having a default built-in expression syntax), but by simplifying
>> > other things it would save a heck of a lot of work (including design
>> > work).
>>
>> So, I'm not sure I follow what exactly are you proposing. To use
>> vanilla Groovy (the actual implementation) for expressions? How does
>> that blend with the directives (which mostly also have similar pairs
>> in Groovy), and other (planned) FM features? (And... why's the result
>> called FreeMarker at all? ;) )
>>
>> > -David
>> >
>> >
>> >
>> > On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <
>> slidingfilaments@gmail.com
>> >> wrote:
>> >
>> >> I'm not a subject matter expert here, but I always thought sensible
>> >> defaults with the ability to override using configuration is a wise
>> >> move because you appeal to many people out of the box and then allow
>> >> minority to also not feel locked out when they need to override. Also,
>> >> as a general rule, I found from the projects we worked on that
>> >> internationalization is always a challenge, but you can also resolve
>> >> that with the same solution (sensible defaults with configuration
>> >> overrides); then you simply let the community adopt and improve the
>> >> code for non-English parts.
>> >>
>> >> Of course then the challenge becomes, what's a sensible default? I
>> >> wish I had a dollar every time I asked that! I'd favor simplicity
>> >> where possible. In your example above maybe "No space f..." is the
>> >> simplest, and then people can get fancy if they want to?
>> >>
>> >> My 2 cents, sorry for the noise :)
>> >>
>> >> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <dd...@apache.org>
>> >> wrote:
>> >> > There's this old frequently requested built-in,
>> >> > maybeTooLongText?truncate(n). Like "No space for such a long
>> >> > title"?truncate(13) would give "No space...". The problem with these
>> >> > is that there's no single algorithm for this that fits the need of
>> >> > everyone. Like maybe the result should be "No space f...", or "No
>> >> > space [...]", etc. This can even change depending on the current value
>> >> > of the `locale` FreeMarker setting.
>> >> >
>> >> > Now there's another similar wish, for "?title_case", which has the
>> >> > same problem, plus it's much more locale dependent. See:
>> >> >
>> >> > https://issues.apache.org/jira/browse/FREEMARKER-97
>> >> >
>> >> > We do provide locale dependent functions, like number and date
>> >> > formatting. This is pretty much an FM tradition, to treat non-English
>> >> > users as equals (well, except that the keywords aren't localizable).
>> >> > However, we push those tasks to the Java platform. But it doesn't
>> >> > support title case (as far as I know). Besides there can be different
>> >> > preferences even inside the same country, just as with ?truncate.
>> >> >
>> >> > So, it's acceptable to chicken out ;) saying that none of the above
>> >> > has a widely accepted default, so the correct solution is letting the
>> >> > user solve this. But I think in practice what happens is that people
>> >> > need the functionality (especially ?truncate), and then has to come up
>> >> > with some ad-hoc solution on the spot, which will be almost always
>> >> > worse than the default we could provide. So we might as well just
>> >> > provide some decent default, sparing time for lot of users. Then, then
>> >> > if they are particular about the algorithm (which is certainly rare
>> >> > nowadays), let them plug their own via the Configurable API. So there
>> >> > would be something like
>> >> > cfg.setTruncateImplementation(localToImplementationMap), where the
>> >> > parameter maps locale-s to Java functions and defines a fallback
>> >> > function. (I haven't worked out the API details, as you can see.)
>> >> >
>> >> > What do you think?
>> >> >
>> >> > --
>> >> > Thanks,
>> >> >  Daniel Dekany
>> >> >
>> >>
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>

-- 
Thanks,
 Daniel Dekany

Re: The problem of "too human" text transformations

[David E Jones <de...@dejc.com>]

You make good points Daniel. At a high level it reminds me of a discussion
with a new client a couple months ago on the topic of managing templates
for ecommerce sites. They want to use the Liquid template engine partly
because they are doing custom ecommerce for some customers and Shopify
(which uses Liquid, where Liquid came from) for others, and also because
Liquid is more constrained and easier for web developers/designers to use.

There is a case for template languages that are less developer oriented and
more designer oriented, or maybe better put because visual web designers
have no business mucking around with templates server side developer vs
web/browser developer (even if this is stuff that generally runs on a
server anyway).

Right now, ie FM2, FreeMarker strikes a good balance between these (IMO
anyway), and the built-ins are an important part of that. Maybe they don't
do every little thing everyone in the world might want but they cover a lot
of basics and common needs.

At a lower level you are correct about template parsing and supporting
arbitrary languages for inline expressions. The template parser still needs
to be able to figure out where the expression starts and stops. This might
not be too bad for C-like languages which current FM expressions are (more
or less, mostly the same demarcation characters and all), but is still a
challenge and could require something like surrounding expressions with
parenthesis (along with nested parenthesis evaluation that FM already does)
and that's a little bit ugly.

Anyway, sorry to derail the discussion.

On the truncate built-in an optional String type second argument makes
sense as a suffix to use if the string is truncated, ie that seems like an
easy thing to accommodate without reducing flexibility or having to write
that logic to do that inline in the template.

On title casing it might be best to defer to what Java does (ie
Character.toUpperCase() or .toTitleCase()), or were there issues with that?

-David


On Thu, May 31, 2018 at 2:03 AM, Daniel Dekany <dd...@apache.org> wrote:

> Thursday, May 31, 2018, 1:16:26 AM, David E Jones wrote:
>
> > My general opinion on this sort of thing is there are lots of libraries
> and
> > tools around in the Java world for doing string manipulations and such,
> > they don't need to be built into a template language. Taking that one
> step
> > further, I'd actually prefer that the expression syntax in the template
> > language makes it easy to call an existing API vs trying to make
> everything
> > built into the template language.
> >
> > For example in addition to using FTL I also use Groovy heavily and Groovy
> > already has a great API including standard Java API augmentation for all
> > sorts of things. Yes, FTL could use the Groovy String and other expanded
> > objects as a reference for built in design and have a good set of
> commonly
> > needed things, but why not just defer to Groovy for those that use it or
> > whatever other API/library anyone might prefer.
> >
> > I know this is a VERY different direction from FM2. What I find most
> useful
> > in FTL is features like macros and the extensive directives and other
> > things that make a template language flexible and powerful. The built-ins
> > are handy sometimes, but often don't do what I need anyway so other than
> > things like ?has_content I don't use them a lot.
>
> An important (and as far as I'm concerned, the main) reason for many
> built-ins is/was that frameworks and other products drop in FreeMarker
> for templates and then often don't give much attention to give tools
> to the template authors. Then the poor people who actually has to
> create templates come and ask, "I must do X... how can I do that in
> FreeMaker?". The less often we have to say "Please pester the
> developers (who are often from another company, or are on another
> sub-project by now => you have near 0 chance) to expose such
> functionality to the templates", the better. Instead we can say "Like
> foo?x", and furthermore, ?x is documented, and people can actually
> find its documentation, even years later. Surely if you have some
> small company with the sharp people, this is not a problem for you.
> Actually, then you very often don't want the one-size-fits-all
> solutions anyway.
>
> > I've mentioned this before but what would be a killer feature for FTL is
> to
> > support pluggable expression evaluators. The FTL expression syntax isn't
> > awful
>
> (Yes it is... but that's besides the point until I have fixed it in
> FM3.)
>
> > but is not nearly as convenient as Groovy, especially when you are
> > using it extensively elsewhere.
>
> The theoretical problem is of course that both languages (the "outer"
> template language and the language from where you rip off the
> expression language from) have its own ideas, and so there will be
> mismatches. The expression language in a language and other parts of
> the language aren't separate in general anyway. Which leads to the
> question that if you are using Groovy expressions, why not also use
> for loops and whatnot from Groovy? (This is fairly common way of
> creating template language, just taking an existing language, and add
> some convenience syntax to embed it into the static text.) So then we
> don't need our own #list, #if, etc. implementation either. But then,
> the original idea was that a template language is a domain specific
> language, and as such can have more specialized, say, loop constructs
> for that domain. (I'm not saying that FM2 did great there, but there
> was a lack of determination there as well.) If that's not the case
> though...
>
> Anyway, when and if FM3 will be more mature, we can see if how doable
> this is in reality. It's also a question if we are talking about a
> syntax, that in 99% looks like Groovy, or we are talking about actual
> Groovy. In the last case TemplateModel-s and the particular way null
> handling will work will be an obstacle for sure. Not to mention
> TemplateMarkupOutputModel-s, that AFAIK don't even have commonly used
> equivalent (like a class in the standard Java API).
>
> > If the default FTL expressions were not used you couldn't use
> > built-ins and instead would just have expressions that are expected
> > to evaluate to a boolean, string, or arbitrary object and can be
> > whatever.
>
> At least in FM3 built-ins will be just functions you have defined in
> the "core" FTL namespace (whose content is specified in template
> language Dialect you use) that's "statically linked" to the templates
> (allowing some parse-time checks and somewhat faster calls). So
> `foo?bar(1)` is basically just a shorthand syntax for `<#import core =
> 'freemarker://dialect-or-something'>` and `core.bar(foo, 1)`. There's
> a standard/default Dialect of course, which has the big advantage that
> it's documented in the official Manual, is used in SO answers, etc.
>
> > How important is this? Important enough that I've
> > considered using a different template language. If FTL wasn't so
> > darn convenience for XML transformation using macros I'd probably be
> > using Groovy GString templates instead, and just for the far better
> > syntax in Groovy.
> >
> > This also goes back to the object wrapping discussion which is another
> can
> > of worms...
>
> Yeah, it's one of the things that I really wanted to get rid of, but
> it's a very disrupting transition to do. I don't mean for most of the
> users, but for the source code. On the brighter side, what really
> happens when you do that transition is that instead of something like:
>
>   if (value is TemplateMapLikeModel) {
>       ... ((TemplateMapLikeModel) value).get(k) ...
>   }
>
> you, in first approach, rewrite it to something like:
>
>   if (mop.isMapLike(value)) {
>       ... mop.get(value, k) ...
>   }
>
> Of course that would be a silly MOP API design, so this below is more
> likely, but now it's visible that one comes from the other trivially:
>
>
>   MapLike m = mop.tryGetAsMapLike(value);
>   if (m != null) {
>       ... m.get(k) ...
>   }
>
> Hence when we have our TemplateModel-s cleaned up, it's easier to
> migrate to the MOP approach. The "type system" (just a mental model
> really) depicted be the TemplateModel still exists, you just use a MOP
> to implement it.
>
> > I don't mean to derail this discussion, but especially these days in a
> > world of a huge number of great Java libraries out there it is often
> better
> > to not try to do everything and instead focus on core strengths that you
> > can't get anywhere else, and make it easy for people to use other open
> > source projects for their core strengths. This could even go as far as
> > FreeMarker builds that have dependencies on certain commonly used open
> > source projects, there are many even in the ASF that do various things
> that
> > FTL built-ins do but with far more options and well known behavior for
> the
> > many who use them (like various Commons projects).
>
> It's really the duty of the Dialects feature of FM3. You can put
> together your "toolbox" from whatever you want, like call Apache
> Commons. I mean it's unrelated to question of expression languages, as
> far as I see. (Like, you can call plain Java methods from FM2 as
> well.)
>
> If we want to use Apache Commons etc. in implementing the functions of
> the standard dialect... I *guess* we won't benefit from it much at
> this point, but I need to see it case-by-case to tell, like, the FM
> implementation of function x sucks, while Apache Commons StringUtils.x
> is great, etc. In any case, users don't care, as far as they can call
> ?x and it does its job. My point in this thread is that we should be
> able to tell users to "just write ?x" (see earlier), instead of add
> your own solution somehow. (Also, if ?x works, they prefer if we have
> less dependencies. Especially if we want to support Andorid.)
>
> > Would supporting different expression languages cause huge problems for
> IDE
> > support? Possibly, but IDEs these days have lots of support for language
> > injection and layers that are meant for just this sort of thing.
>
> At least looking from big enough distance it's easy. :) Note that so
> far we fail to deliver an editor for Eclipse that integrates the
> existing HTML editor functionality with FTL... :-> Please everyone go
> and help out Angelo Zerr so that this will change:
>
> https://github.com/angelozerr/lsp4e-freemarker
> https://github.com/angelozerr/freemarker-languageserver
>
> Anyway, IDE support won't be the obstacle. At worse only the default
> expression syntax will have reasonable IDE support.
>
> > In theory FreeMarker itself could be a heck of a lot smaller and
> > simpler and at the same time be far more flexible and powerful. IMO
> > that could be a game changer in the world of Java-based template
> > languages. Some of this would be more work than not supporting this
> > sort of flexibility (ie support Groovy syntax as an option, ie
> > without requiring the much larger Groovy jar files as dependencies
> > by having a default built-in expression syntax), but by simplifying
> > other things it would save a heck of a lot of work (including design
> > work).
>
> So, I'm not sure I follow what exactly are you proposing. To use
> vanilla Groovy (the actual implementation) for expressions? How does
> that blend with the directives (which mostly also have similar pairs
> in Groovy), and other (planned) FM features? (And... why's the result
> called FreeMarker at all? ;) )
>
> > -David
> >
> >
> >
> > On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <
> slidingfilaments@gmail.com
> >> wrote:
> >
> >> I'm not a subject matter expert here, but I always thought sensible
> >> defaults with the ability to override using configuration is a wise
> >> move because you appeal to many people out of the box and then allow
> >> minority to also not feel locked out when they need to override. Also,
> >> as a general rule, I found from the projects we worked on that
> >> internationalization is always a challenge, but you can also resolve
> >> that with the same solution (sensible defaults with configuration
> >> overrides); then you simply let the community adopt and improve the
> >> code for non-English parts.
> >>
> >> Of course then the challenge becomes, what's a sensible default? I
> >> wish I had a dollar every time I asked that! I'd favor simplicity
> >> where possible. In your example above maybe "No space f..." is the
> >> simplest, and then people can get fancy if they want to?
> >>
> >> My 2 cents, sorry for the noise :)
> >>
> >> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <dd...@apache.org>
> >> wrote:
> >> > There's this old frequently requested built-in,
> >> > maybeTooLongText?truncate(n). Like "No space for such a long
> >> > title"?truncate(13) would give "No space...". The problem with these
> >> > is that there's no single algorithm for this that fits the need of
> >> > everyone. Like maybe the result should be "No space f...", or "No
> >> > space [...]", etc. This can even change depending on the current value
> >> > of the `locale` FreeMarker setting.
> >> >
> >> > Now there's another similar wish, for "?title_case", which has the
> >> > same problem, plus it's much more locale dependent. See:
> >> >
> >> > https://issues.apache.org/jira/browse/FREEMARKER-97
> >> >
> >> > We do provide locale dependent functions, like number and date
> >> > formatting. This is pretty much an FM tradition, to treat non-English
> >> > users as equals (well, except that the keywords aren't localizable).
> >> > However, we push those tasks to the Java platform. But it doesn't
> >> > support title case (as far as I know). Besides there can be different
> >> > preferences even inside the same country, just as with ?truncate.
> >> >
> >> > So, it's acceptable to chicken out ;) saying that none of the above
> >> > has a widely accepted default, so the correct solution is letting the
> >> > user solve this. But I think in practice what happens is that people
> >> > need the functionality (especially ?truncate), and then has to come up
> >> > with some ad-hoc solution on the spot, which will be almost always
> >> > worse than the default we could provide. So we might as well just
> >> > provide some decent default, sparing time for lot of users. Then, then
> >> > if they are particular about the algorithm (which is certainly rare
> >> > nowadays), let them plug their own via the Configurable API. So there
> >> > would be something like
> >> > cfg.setTruncateImplementation(localToImplementationMap), where the
> >> > parameter maps locale-s to Java functions and defines a fallback
> >> > function. (I haven't worked out the API details, as you can see.)
> >> >
> >> > What do you think?
> >> >
> >> > --
> >> > Thanks,
> >> >  Daniel Dekany
> >> >
> >>
>
> --
> Thanks,
>  Daniel Dekany
>
>

Re: The problem of "too human" text transformations

[Daniel Dekany <dd...@apache.org>]

Thursday, May 31, 2018, 1:16:26 AM, David E Jones wrote:

> My general opinion on this sort of thing is there are lots of libraries and
> tools around in the Java world for doing string manipulations and such,
> they don't need to be built into a template language. Taking that one step
> further, I'd actually prefer that the expression syntax in the template
> language makes it easy to call an existing API vs trying to make everything
> built into the template language.
>
> For example in addition to using FTL I also use Groovy heavily and Groovy
> already has a great API including standard Java API augmentation for all
> sorts of things. Yes, FTL could use the Groovy String and other expanded
> objects as a reference for built in design and have a good set of commonly
> needed things, but why not just defer to Groovy for those that use it or
> whatever other API/library anyone might prefer.
>
> I know this is a VERY different direction from FM2. What I find most useful
> in FTL is features like macros and the extensive directives and other
> things that make a template language flexible and powerful. The built-ins
> are handy sometimes, but often don't do what I need anyway so other than
> things like ?has_content I don't use them a lot.

An important (and as far as I'm concerned, the main) reason for many
built-ins is/was that frameworks and other products drop in FreeMarker
for templates and then often don't give much attention to give tools
to the template authors. Then the poor people who actually has to
create templates come and ask, "I must do X... how can I do that in
FreeMaker?". The less often we have to say "Please pester the
developers (who are often from another company, or are on another
sub-project by now => you have near 0 chance) to expose such
functionality to the templates", the better. Instead we can say "Like
foo?x", and furthermore, ?x is documented, and people can actually
find its documentation, even years later. Surely if you have some
small company with the sharp people, this is not a problem for you.
Actually, then you very often don't want the one-size-fits-all
solutions anyway.

> I've mentioned this before but what would be a killer feature for FTL is to
> support pluggable expression evaluators. The FTL expression syntax isn't
> awful

(Yes it is... but that's besides the point until I have fixed it in
FM3.)

> but is not nearly as convenient as Groovy, especially when you are
> using it extensively elsewhere.

The theoretical problem is of course that both languages (the "outer"
template language and the language from where you rip off the
expression language from) have its own ideas, and so there will be
mismatches. The expression language in a language and other parts of
the language aren't separate in general anyway. Which leads to the
question that if you are using Groovy expressions, why not also use
for loops and whatnot from Groovy? (This is fairly common way of
creating template language, just taking an existing language, and add
some convenience syntax to embed it into the static text.) So then we
don't need our own #list, #if, etc. implementation either. But then,
the original idea was that a template language is a domain specific
language, and as such can have more specialized, say, loop constructs
for that domain. (I'm not saying that FM2 did great there, but there
was a lack of determination there as well.) If that's not the case
though...

Anyway, when and if FM3 will be more mature, we can see if how doable
this is in reality. It's also a question if we are talking about a
syntax, that in 99% looks like Groovy, or we are talking about actual
Groovy. In the last case TemplateModel-s and the particular way null
handling will work will be an obstacle for sure. Not to mention
TemplateMarkupOutputModel-s, that AFAIK don't even have commonly used
equivalent (like a class in the standard Java API).

> If the default FTL expressions were not used you couldn't use
> built-ins and instead would just have expressions that are expected
> to evaluate to a boolean, string, or arbitrary object and can be
> whatever.

At least in FM3 built-ins will be just functions you have defined in
the "core" FTL namespace (whose content is specified in template
language Dialect you use) that's "statically linked" to the templates
(allowing some parse-time checks and somewhat faster calls). So
`foo?bar(1)` is basically just a shorthand syntax for `<#import core =
'freemarker://dialect-or-something'>` and `core.bar(foo, 1)`. There's
a standard/default Dialect of course, which has the big advantage that
it's documented in the official Manual, is used in SO answers, etc.

> How important is this? Important enough that I've
> considered using a different template language. If FTL wasn't so
> darn convenience for XML transformation using macros I'd probably be
> using Groovy GString templates instead, and just for the far better
> syntax in Groovy.
>
> This also goes back to the object wrapping discussion which is another can
> of worms...

Yeah, it's one of the things that I really wanted to get rid of, but
it's a very disrupting transition to do. I don't mean for most of the
users, but for the source code. On the brighter side, what really
happens when you do that transition is that instead of something like:

  if (value is TemplateMapLikeModel) {
      ... ((TemplateMapLikeModel) value).get(k) ...
  }

you, in first approach, rewrite it to something like:

  if (mop.isMapLike(value)) {
      ... mop.get(value, k) ...
  }

Of course that would be a silly MOP API design, so this below is more
likely, but now it's visible that one comes from the other trivially:


  MapLike m = mop.tryGetAsMapLike(value);
  if (m != null) {
      ... m.get(k) ...
  }

Hence when we have our TemplateModel-s cleaned up, it's easier to
migrate to the MOP approach. The "type system" (just a mental model
really) depicted be the TemplateModel still exists, you just use a MOP
to implement it.

> I don't mean to derail this discussion, but especially these days in a
> world of a huge number of great Java libraries out there it is often better
> to not try to do everything and instead focus on core strengths that you
> can't get anywhere else, and make it easy for people to use other open
> source projects for their core strengths. This could even go as far as
> FreeMarker builds that have dependencies on certain commonly used open
> source projects, there are many even in the ASF that do various things that
> FTL built-ins do but with far more options and well known behavior for the
> many who use them (like various Commons projects).

It's really the duty of the Dialects feature of FM3. You can put
together your "toolbox" from whatever you want, like call Apache
Commons. I mean it's unrelated to question of expression languages, as
far as I see. (Like, you can call plain Java methods from FM2 as
well.)

If we want to use Apache Commons etc. in implementing the functions of
the standard dialect... I *guess* we won't benefit from it much at
this point, but I need to see it case-by-case to tell, like, the FM
implementation of function x sucks, while Apache Commons StringUtils.x
is great, etc. In any case, users don't care, as far as they can call
?x and it does its job. My point in this thread is that we should be
able to tell users to "just write ?x" (see earlier), instead of add
your own solution somehow. (Also, if ?x works, they prefer if we have
less dependencies. Especially if we want to support Andorid.)

> Would supporting different expression languages cause huge problems for IDE
> support? Possibly, but IDEs these days have lots of support for language
> injection and layers that are meant for just this sort of thing.

At least looking from big enough distance it's easy. :) Note that so
far we fail to deliver an editor for Eclipse that integrates the
existing HTML editor functionality with FTL... :-> Please everyone go
and help out Angelo Zerr so that this will change:

https://github.com/angelozerr/lsp4e-freemarker
https://github.com/angelozerr/freemarker-languageserver

Anyway, IDE support won't be the obstacle. At worse only the default
expression syntax will have reasonable IDE support.

> In theory FreeMarker itself could be a heck of a lot smaller and
> simpler and at the same time be far more flexible and powerful. IMO
> that could be a game changer in the world of Java-based template
> languages. Some of this would be more work than not supporting this
> sort of flexibility (ie support Groovy syntax as an option, ie
> without requiring the much larger Groovy jar files as dependencies
> by having a default built-in expression syntax), but by simplifying
> other things it would save a heck of a lot of work (including design
> work).

So, I'm not sure I follow what exactly are you proposing. To use
vanilla Groovy (the actual implementation) for expressions? How does
that blend with the directives (which mostly also have similar pairs
in Groovy), and other (planned) FM features? (And... why's the result
called FreeMarker at all? ;) )

> -David
>
>
>
> On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <slidingfilaments@gmail.com
>> wrote:
>
>> I'm not a subject matter expert here, but I always thought sensible
>> defaults with the ability to override using configuration is a wise
>> move because you appeal to many people out of the box and then allow
>> minority to also not feel locked out when they need to override. Also,
>> as a general rule, I found from the projects we worked on that
>> internationalization is always a challenge, but you can also resolve
>> that with the same solution (sensible defaults with configuration
>> overrides); then you simply let the community adopt and improve the
>> code for non-English parts.
>>
>> Of course then the challenge becomes, what's a sensible default? I
>> wish I had a dollar every time I asked that! I'd favor simplicity
>> where possible. In your example above maybe "No space f..." is the
>> simplest, and then people can get fancy if they want to?
>>
>> My 2 cents, sorry for the noise :)
>>
>> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <dd...@apache.org>
>> wrote:
>> > There's this old frequently requested built-in,
>> > maybeTooLongText?truncate(n). Like "No space for such a long
>> > title"?truncate(13) would give "No space...". The problem with these
>> > is that there's no single algorithm for this that fits the need of
>> > everyone. Like maybe the result should be "No space f...", or "No
>> > space [...]", etc. This can even change depending on the current value
>> > of the `locale` FreeMarker setting.
>> >
>> > Now there's another similar wish, for "?title_case", which has the
>> > same problem, plus it's much more locale dependent. See:
>> >
>> > https://issues.apache.org/jira/browse/FREEMARKER-97
>> >
>> > We do provide locale dependent functions, like number and date
>> > formatting. This is pretty much an FM tradition, to treat non-English
>> > users as equals (well, except that the keywords aren't localizable).
>> > However, we push those tasks to the Java platform. But it doesn't
>> > support title case (as far as I know). Besides there can be different
>> > preferences even inside the same country, just as with ?truncate.
>> >
>> > So, it's acceptable to chicken out ;) saying that none of the above
>> > has a widely accepted default, so the correct solution is letting the
>> > user solve this. But I think in practice what happens is that people
>> > need the functionality (especially ?truncate), and then has to come up
>> > with some ad-hoc solution on the spot, which will be almost always
>> > worse than the default we could provide. So we might as well just
>> > provide some decent default, sparing time for lot of users. Then, then
>> > if they are particular about the algorithm (which is certainly rare
>> > nowadays), let them plug their own via the Configurable API. So there
>> > would be something like
>> > cfg.setTruncateImplementation(localToImplementationMap), where the
>> > parameter maps locale-s to Java functions and defines a fallback
>> > function. (I haven't worked out the API details, as you can see.)
>> >
>> > What do you think?
>> >
>> > --
>> > Thanks,
>> >  Daniel Dekany
>> >
>>

-- 
Thanks,
 Daniel Dekany

Re: The problem of "too human" text transformations

[David E Jones <de...@dejc.com>]

My general opinion on this sort of thing is there are lots of libraries and
tools around in the Java world for doing string manipulations and such,
they don't need to be built into a template language. Taking that one step
further, I'd actually prefer that the expression syntax in the template
language makes it easy to call an existing API vs trying to make everything
built into the template language.

For example in addition to using FTL I also use Groovy heavily and Groovy
already has a great API including standard Java API augmentation for all
sorts of things. Yes, FTL could use the Groovy String and other expanded
objects as a reference for built in design and have a good set of commonly
needed things, but why not just defer to Groovy for those that use it or
whatever other API/library anyone might prefer.

I know this is a VERY different direction from FM2. What I find most useful
in FTL is features like macros and the extensive directives and other
things that make a template language flexible and powerful. The built-ins
are handy sometimes, but often don't do what I need anyway so other than
things like ?has_content I don't use them a lot.

I've mentioned this before but what would be a killer feature for FTL is to
support pluggable expression evaluators. The FTL expression syntax isn't
awful but is not nearly as convenient as Groovy, especially when you are
using it extensively elsewhere. If the default FTL expressions were not
used you couldn't use built-ins and instead would just have expressions
that are expected to evaluate to a boolean, string, or arbitrary object and
can be whatever. How important is this? Important enough that I've
considered using a different template language. If FTL wasn't so darn
convenience for XML transformation using macros I'd probably be using
Groovy GString templates instead, and just for the far better syntax in
Groovy.

This also goes back to the object wrapping discussion which is another can
of worms...

I don't mean to derail this discussion, but especially these days in a
world of a huge number of great Java libraries out there it is often better
to not try to do everything and instead focus on core strengths that you
can't get anywhere else, and make it easy for people to use other open
source projects for their core strengths. This could even go as far as
FreeMarker builds that have dependencies on certain commonly used open
source projects, there are many even in the ASF that do various things that
FTL built-ins do but with far more options and well known behavior for the
many who use them (like various Commons projects).

Would supporting different expression languages cause huge problems for IDE
support? Possibly, but IDEs these days have lots of support for language
injection and layers that are meant for just this sort of thing.

In theory FreeMarker itself could be a heck of a lot smaller and simpler
and at the same time be far more flexible and powerful. IMO that could be a
game changer in the world of Java-based template languages. Some of this
would be more work than not supporting this sort of flexibility (ie support
Groovy syntax as an option, ie without requiring the much larger Groovy jar
files as dependencies by having a default built-in expression syntax), but
by simplifying other things it would save a heck of a lot of work
(including design work).

-David



On Wed, May 30, 2018 at 5:50 AM, Taher Alkhateeb <slidingfilaments@gmail.com
> wrote:

> I'm not a subject matter expert here, but I always thought sensible
> defaults with the ability to override using configuration is a wise
> move because you appeal to many people out of the box and then allow
> minority to also not feel locked out when they need to override. Also,
> as a general rule, I found from the projects we worked on that
> internationalization is always a challenge, but you can also resolve
> that with the same solution (sensible defaults with configuration
> overrides); then you simply let the community adopt and improve the
> code for non-English parts.
>
> Of course then the challenge becomes, what's a sensible default? I
> wish I had a dollar every time I asked that! I'd favor simplicity
> where possible. In your example above maybe "No space f..." is the
> simplest, and then people can get fancy if they want to?
>
> My 2 cents, sorry for the noise :)
>
> On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <dd...@apache.org>
> wrote:
> > There's this old frequently requested built-in,
> > maybeTooLongText?truncate(n). Like "No space for such a long
> > title"?truncate(13) would give "No space...". The problem with these
> > is that there's no single algorithm for this that fits the need of
> > everyone. Like maybe the result should be "No space f...", or "No
> > space [...]", etc. This can even change depending on the current value
> > of the `locale` FreeMarker setting.
> >
> > Now there's another similar wish, for "?title_case", which has the
> > same problem, plus it's much more locale dependent. See:
> >
> > https://issues.apache.org/jira/browse/FREEMARKER-97
> >
> > We do provide locale dependent functions, like number and date
> > formatting. This is pretty much an FM tradition, to treat non-English
> > users as equals (well, except that the keywords aren't localizable).
> > However, we push those tasks to the Java platform. But it doesn't
> > support title case (as far as I know). Besides there can be different
> > preferences even inside the same country, just as with ?truncate.
> >
> > So, it's acceptable to chicken out ;) saying that none of the above
> > has a widely accepted default, so the correct solution is letting the
> > user solve this. But I think in practice what happens is that people
> > need the functionality (especially ?truncate), and then has to come up
> > with some ad-hoc solution on the spot, which will be almost always
> > worse than the default we could provide. So we might as well just
> > provide some decent default, sparing time for lot of users. Then, then
> > if they are particular about the algorithm (which is certainly rare
> > nowadays), let them plug their own via the Configurable API. So there
> > would be something like
> > cfg.setTruncateImplementation(localToImplementationMap), where the
> > parameter maps locale-s to Java functions and defines a fallback
> > function. (I haven't worked out the API details, as you can see.)
> >
> > What do you think?
> >
> > --
> > Thanks,
> >  Daniel Dekany
> >
>

Re: The problem of "too human" text transformations

[Taher Alkhateeb <sl...@gmail.com>]

I'm not a subject matter expert here, but I always thought sensible
defaults with the ability to override using configuration is a wise
move because you appeal to many people out of the box and then allow
minority to also not feel locked out when they need to override. Also,
as a general rule, I found from the projects we worked on that
internationalization is always a challenge, but you can also resolve
that with the same solution (sensible defaults with configuration
overrides); then you simply let the community adopt and improve the
code for non-English parts.

Of course then the challenge becomes, what's a sensible default? I
wish I had a dollar every time I asked that! I'd favor simplicity
where possible. In your example above maybe "No space f..." is the
simplest, and then people can get fancy if they want to?

My 2 cents, sorry for the noise :)

On Wed, May 30, 2018 at 11:59 AM, Daniel Dekany <dd...@apache.org> wrote:
> There's this old frequently requested built-in,
> maybeTooLongText?truncate(n). Like "No space for such a long
> title"?truncate(13) would give "No space...". The problem with these
> is that there's no single algorithm for this that fits the need of
> everyone. Like maybe the result should be "No space f...", or "No
> space [...]", etc. This can even change depending on the current value
> of the `locale` FreeMarker setting.
>
> Now there's another similar wish, for "?title_case", which has the
> same problem, plus it's much more locale dependent. See:
>
> https://issues.apache.org/jira/browse/FREEMARKER-97
>
> We do provide locale dependent functions, like number and date
> formatting. This is pretty much an FM tradition, to treat non-English
> users as equals (well, except that the keywords aren't localizable).
> However, we push those tasks to the Java platform. But it doesn't
> support title case (as far as I know). Besides there can be different
> preferences even inside the same country, just as with ?truncate.
>
> So, it's acceptable to chicken out ;) saying that none of the above
> has a widely accepted default, so the correct solution is letting the
> user solve this. But I think in practice what happens is that people
> need the functionality (especially ?truncate), and then has to come up
> with some ad-hoc solution on the spot, which will be almost always
> worse than the default we could provide. So we might as well just
> provide some decent default, sparing time for lot of users. Then, then
> if they are particular about the algorithm (which is certainly rare
> nowadays), let them plug their own via the Configurable API. So there
> would be something like
> cfg.setTruncateImplementation(localToImplementationMap), where the
> parameter maps locale-s to Java functions and defines a fallback
> function. (I haven't worked out the API details, as you can see.)
>
> What do you think?
>
> --
> Thanks,
>  Daniel Dekany
>