Re : Re: Lambda Expressions - filter list without <#list> directive

[Denis Bredelet <br...@mac.com.INVALID>]

Hi,


Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com> a écrit :


Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <ddekany@apache.org

:


It's certainly tricky, but as far as I see possible (but then, who

knows what will one find when actually working on it). It's also a

feature missing a lot. It's especially missing for #list (I know that
you need it for something else), because if you filter the items
inside #list with #if-s, then #sep, ?hasNext, etc. will not be usable.


Let me say that I disagree here.



I do not think that closures are required for FreeMarker, nor that they are a good idea.



If we add new features to the FreeMarker *tempate engine* I would rather we focus on multi-part macro body rather than an advanced language feature like closures.



You can add ?filter and ?map if you want, a simple expression as parameter should be enough.



Cheers,

-- Denis.

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

Saturday, February 23, 2019, 6:40:52 PM, Denis Bredelet wrote:

> Hi Daniel,
>
>> Wow, it's amazing!
>> I'd spend some time testing it.
>> 
>> Thanks a lot!
>
> Agreed, that looks wonderful. I want to test it.
>
> Another use could be “combine” which applies a lambda to the list and one more list (zipping):
>
> <#list products?combine(links, (p, l) -> linkify(p.name, l)) as clickableProduct>
> ...
>
> Not sure I like the syntax I came up with. Maybe this would be better:
>
>   products?combine(links)?map(p, l -> linkify(p.name, l))
>
> Will that be useful though?

Did you run into this situation, and if so, often? I'm curious about
the use case then, and if how much was it because of badly designed
data model. So, your are saying that collection1?combine(collection2,
(elementFromCollection1, elementFromcollection2) -> ...) pairs the
elements of the two collections by their common index, and then they
are combined. But how often are there such collections in the
data-model? It's like if you have a table, and instead of having a
List of row objects, where each row object contains the values for all
the columns, you have a List for each column, where each element only
stores the value for that single column. Looks weird to me. (It's like
columnar data storage, but that's a low level technique, which is
certainly not exposed to templates...)

> — Denis.
>
>> Woonsan
>> 
>> On Sat, Feb 23, 2019 at 12:19 PM Daniel Dekany <dd...@apache.org> wrote:
>>> 
>>> I have pushed an implementation of the restricted lambdas we were
>>> talking about. Guys, please review/test it.
>>> 
>>> I call these "local lambdas" in the source code (but I'm open for
>>> suggestions), as the function they define can only be called in the
>>> same variable scope where the lambda was (as we have no closures in
>>> the template language, nor final variables). Also, they can only be
>>> used as the parameters of the built-ins that explicitly support them.
>>> As the subject shows, the main goal was just support filtering for
>>> #list, without a nested #if, as that breaks #sep, #else, it?hasNext,
>>> it?index, etc. (I will try to make more universal lambdas in FM3... I
>>> guess it would be way too tricky in FM2.)
>>> 
>>> For now, I have only added two built-ins that support lambdas: ?filter
>>> and ?map. Any ideas what other such built-ins would be often useful in
>>> templates (with use case, if possible)?
>>> 
>>> Examples of using ?filter and ?map:
>>> 
>>>  <#list products?filter(it -> it.price < 1000) as product>
>>>    ${product.name}
>>>  </#list>
>>> 
>>>  <#list products?map(it -> it.name) as name>
>>>    ${name}
>>>  </#list>
>>> 
>>> Of course these built-ins aren't specific to #list, they can be used
>>> anywhere. Naturally, they can be chained as well:
>>> 
>>>  <#assign chepProdNames = products
>>>     ?filter(it -> it.price < 1000)
>>>     ?map(it -> it.name)
>>>> 
>>> 
>>> As a side note, ?filter and ?map also accepts FTL function-s and Java
>>> methods as its parameter, not only lambdas.
>>> 
>>> A tricky aspect of this feature is lazy evaluation, in similar sense
>>> as Java 8 Stream intermediate operations are lazy. On most places,
>>> ?filer and ?map are eager, that is, they return a completed sequence
>>> of items. That's because our restricted lambdas only work correctly
>>> "locally". However, there are very common situations where it's clear
>>> that we can use lazy ("streaming") evaluation, as we know where the
>>> resulting stream of elements will be consumed:
>>> 
>>> - One such case is when these kind of built-ins are chained. Like in
>>>  the last example, ?filter doesn't construct a List in memory,
>>>  instead the elements just flow through it (some is dropped, as it's
>>>  filtering), into ?map. Only the ?map at the end of the chain will
>>>  built a List eagerly. Some other built-ins also allow the left-hand
>>>  built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
>>>  is built anywhere, instead the elements flow through both ?filter
>>>  and ?map, and ?join just appends them to the StringBuilder where it
>>>  creates its results.
>>> 
>>> - #list also enables lazy evaluation to its 1st parameter. So in the
>>>  #list examples above, yet again no List is built in memory.
>>> 
>>> - There are other such cases, which I didn't implement yet. For
>>>  example the sequence slice operator, like, xs?filter(f)[10..20],
>>>  should allow lazy evaluation.
>>> 
>>> Feedback is welcome!
>>> 
>>> 
>>> Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
>>> 
>>>> Hey Daniel,
>>>> I'm very sorry, but I didn't make any progress with this. I think I was a
>>>> bit over-motivated, but unfortunately I cannot spend more time on this for
>>>> various reasons.
>>>> You can take this over. I'm glad to help out with testing and feedback.
>>>> 
>>>> Thanks
>>>> Christoph
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>>>> :
>>>> 
>>>>> Any progress in this? I think I will give it a try in the coming days
>>>>> otherwise.
>>>>> 
>>>>> 
>>>>> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
>>>>> 
>>>>>> See my answers inline...
>>>>>> 
>>>>>> Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
>>>>>> 
>>>>>>> Thanks Daniel for your feedback. See my answers below
>>>>>>> 
>>>>>>> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
>>>>> ddekany@apache.org
>>>>>>>> :
>>>>>>> 
>>>>>>>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>>>>>>>> 
>>>>>>>>> Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
>>>>>>>> ddekany@apache.org
>>>>>>>>>> :
>>>>>>>>> 
>>>>>>>>>> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com>
>>>>> a
>>>>>>>>>> écrit :
>>>>>>>>>>> 
>>>>>>>>>>> Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>>>>>>>>>> ddekany@apache.org
>>>>>>>>>>> :
>>>>>>>>>>> 
>>>>>>>>>>> It's certainly tricky, but as far as I see possible (but then, who
>>>>>>>>>>> 
>>>>>>>>>>> knows what will one find when actually working on it). It's also a
>>>>>>>>>>> feature missing a lot. It's especially missing for #list (I know
>>>>> that
>>>>>>>>>>> you need it for something else), because if you filter the items
>>>>>>>>>>> inside #list with #if-s, then #sep, ?hasNext, etc. will not be
>>>>> usable.
>>>>>>>>>>> 
>>>>>>>>>>> Let me say that I disagree here.
>>>>>>>>>>> 
>>>>>>>>>>> I do not think that closures are required for FreeMarker, nor that
>>>>>>>> they
>>>>>>>>>> are a good idea.
>>>>>>>>>>> 
>>>>>>>>>>> If we add new features to the FreeMarker *tempate engine* I would
>>>>>>>>>>> rather we focus on multi-part macro body rather than an advanced
>>>>>>>>>> language feature like closures.
>>>>>>>>>>> 
>>>>>>>>>>> You can add ?filter and ?map if you want, a simple expression as
>>>>>>>>>> parameter should be enough.
>>>>>>>>>> 
>>>>>>>>>> Yes, as I said, we certainly start with only allowing lambdas in
>>>>>>>>>> ?filter/?map, also certainly in ?contains.
>>>>>>>>>> 
>>>>>>>>> Would be enough in my opinion and very useful.
>>>>>>>>> 
>>>>>>>>> Is it possiblefor you to give some pointers to the code on how this
>>>>> could
>>>>>>>>> be implemented? I would maybe like to wrap my head around this a
>>>>> little
>>>>>>>> bit.
>>>>>>>> 
>>>>>>>> Please feel yourself encouraged! (:
>>>>>>>> 
>>>>>>>>> I started looking at seq_containsBI (
>>>>>>>>> 
>>>>>>>> 
>>>>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
>>>>>>>> )
>>>>>>>>> and
>>>>>>>>> and reverseBI (
>>>>>>>>> 
>>>>>>>> 
>>>>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
>>>>>>>> )
>>>>>>>>> just to find something related (seq_containsBI checks something) and
>>>>>>>>> reverseBI returns a new sequence.
>>>>>>>>> What I haven't found is a function which takes an Expression as a
>>>>>>>>> parameter.
>>>>>>>>> Is there something similar already or would that be a new thing?
>>>>>>>> 
>>>>>>>> It's a new thing in that it will be part of the expression syntax
>>>>>>>> (even if for now we will only allow lambdas as the parameters of a few
>>>>>>>> built-ins, so that we can get away without closures). So it's a new
>>>>>>>> Expression subclass, and has to be part of the parser (ftl.jj) as
>>>>>>>> well.
>>>>>>> 
>>>>>>> Hmm, that parser stuff is new for me, it'll take me some time to get
>>>>> into
>>>>>>> it.
>>>>>> 
>>>>>> So it's a JavaCC lexer+parser. With a few twists... sorry, but this
>>>>>> code has history... :)
>>>>>> 
>>>>>>>> As of lazy evaluation of parameters expressions, that's already
>>>>>>>> done in the built-ins in BuiltInsWithParseTimeParameters, and you will
>>>>>>>> see it's trivial to do, but the situation there is much simpler.
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> In principle, a LambdaExpression should evaluate to a
>>>>>>>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
>>>>>>>> the called built-in or whatever it is. But with the approach of
>>>>>>>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
>>>>>>>> just bind to the LambdaExpression directly, add a LocalContext that
>>>>>>>> contains the lambda arguments, end evaluate the LambdaExpression right
>>>>>>>> there, in the built-in implementation. Or at least at a very quick
>>>>>>>> glance I think so.
>>>>>>>> 
>>>>>>> Not sure I can follow completely but that hint with*
>>>>>>> BuiltInsWithParseTimeParameters* got me started, but at the moment I'm
>>>>>>> stuck as I need to get more familiar with the internals of Freemarker.
>>>>> I am
>>>>>>> also not sure I am on the same page regarding the syntax we are aiming
>>>>> for
>>>>>>> and why I would need to extend the parser when there is something like
>>>>>>> BuiltInsWithParseTimeParameters....
>>>>>> 
>>>>>> I assumed that the syntax will be similar to the Java lambda syntax
>>>>>> (see below why), and that's of course needs lexer/parser changes.
>>>>>> 
>>>>>>> Here is an example what I have in mind:
>>>>>>> I started with the ?filter() builtin. I had a syntax like this in mind:
>>>>>>> 
>>>>>>> *Example 1: ["a","b","c"]?filter(element, element == "c")*
>>>>>>> *Example 2: ["a","b","c"]?filter(element, element == someOtherVariable,
>>>>>>> someOtherVariable="c")*
>>>>>> 
>>>>>> Looking at the above one believe that the value of `element` is passed
>>>>>> in as first argument, and the the value of `element == "c"` as the
>>>>>> second, but that's not the case. It's much better if it's visible that
>>>>>> you got some kind of anonymous function definition there without
>>>>>> knowing about the "filter" built-in.
>>>>>> 
>>>>>> Java already has a syntax for expressing this kind of thing, so it
>>>>>> would be better to use that familiar syntax. As far as I know it
>>>>>> doesn't conflict with ours (it kind of it does, but not fatally).
>>>>>> 
>>>>>> Also, even if for now we only allow this in said built-ins, we
>>>>>> shouldn't exclude the possibility of making this kind of expression
>>>>>> accessible elsewhere as well. Then, on many places the parsed won't
>>>>>> know what kind of value is expected (consider passing a lambda to an
>>>>>> user defined directive for example), so a syntax like above won't
>>>>>> work.
>>>>>> 
>>>>>>> Not sure if that's what you have in mind too, but to me it made sense
>>>>> with
>>>>>>> regards to BuiltInsWithParseTimeParameters and I could start without
>>>>>>> touching parser stuff.
>>>>>> 
>>>>>> BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
>>>>>> bit...), as a call to a such built-in looks like a call to any other
>>>>>> built-in.
>>>>>> 
>>>>>>> *1st argument 'element'* would just be the iterator variable similar to
>>>>>>> <#list ["a","b","c"] as *element*>
>>>>>>> 2nd argument is the filter lambda expression... aka our filter condition
>>>>>>> 3rd+n argument are optional parameters in case used in the lambda
>>>>> expression
>>>>>> 
>>>>>> #list is special, similarly as `for ... in ...` is in most languages.
>>>>>> It's not lambda-ish either; you can't write `as element/2` or such.
>>>>>> 
>>>>>>> So at first I was looking how <#list> works and found IteratorBlock, and
>>>>>>> though I could reuse it somehow.
>>>>>>> 
>>>>>>> Here is some simple pseudo code I played around for the for Example 1:
>>>>>>> 
>>>>>>> static class filter_BI extends BuiltInWithParseTimeParameters {
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>        TemplateModel _eval(Environment env) throws TemplateException {
>>>>>>> 
>>>>>>>            // sequence
>>>>>>> 
>>>>>>>        TemplateModel targetValue = target.evalToNonMissing(env);
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>            List parameters = this.parameters;
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>            Expression iteratorAlias = (Expression) parameters.get(0);
>>>>>>> 
>>>>>>>            Expression conditionExpression = (Expression)
>>>>> parameters.get(1);
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>            TemplateSequenceModel seq =  (TemplateSequenceModel)
>>>>> target.eval
>>>>>>> (env);
>>>>>>> 
>>>>>>>            for (int i = 0; i < seq.size(); i++) {
>>>>>>> 
>>>>>>>               TemplateModel cur = seq.get(i);
>>>>>>> 
>>>>>>> 
>>>>>>>               // this is where I am stuck at the moment
>>>>>>> 
>>>>>>>               // I basically want to evaluate conditionExpression
>>>>>>> where iteratorAlias
>>>>>>> is basically what I passed as 'element'
>>>>>>> 
>>>>>>>               // I am not sure if or how LocalContext could come into
>>>>> play
>>>>>>> here
>>>>>>> 
>>>>>>>               // basically for each iteration I would assign the
>>>>> current
>>>>>>> loop element to a context variable with the name 'element'
>>>>>>> 
>>>>>>>               // and then evaluate conditionExpression with that
>>>>> context.
>>>>>>> 
>>>>>>>               // if conditionExpression is "true" then I would populate
>>>>>>> add the current sequence element 'cur'
>>>>>>> 
>>>>>>>               // to a new result-List.... and return that.... something
>>>>>>> 
>>>>>>>               // I wanted to reuse IteratorBlock here somehow, but
>>>>> didn't
>>>>>>> get it to work yet.
>>>>>>> 
>>>>>>>               // maybe this is a stupid idea, or we just need something
>>>>>>> similar
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>            }
>>>>>>> 
>>>>>>> }
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Ok so far for my pseudo code....  Maybe you could give some more
>>>>> pointers
>>>>>>> based on that... in case this makes any sense ...
>>>>>> 
>>>>>> For LocalContext, Environment has a pushLocalContext method. See the
>>>>>> calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
>>>>>> Map, List).
>>>>>> 
>>>>>> To avoid running into more new things at once than necessary, perhaps
>>>>>> you should start with seq?seq_contains(lambda). The end result should
>>>>>> be like:
>>>>>> 
>>>>>>  users?contains(u -> u.paying)
>>>>>> 
>>>>>>>> Another similarity to BuiltInsWithParseTimeParameters is that we won't
>>>>>>>> allow separating the `?someBI` and `(arg)`. Like, with the example of
>>>>>>>> `cond?then(1, 2)`, you aren't allowed to do this:
>>>>>>>> 
>>>>>>>>  <#assign t=cond?then>
>>>>>>>>  ${t(1, 2)}
>>>>>>>> 
>>>>>>>> That maybe looks natural, but most other built-ins allow that. Of
>>>>>>>> course we can't allow that for ?filter etc., because then we need
>>>>>>>> closures again.
>>>>>>>> 
>>>>>>>>>> Multi-part macro body is also planned. Means, I know it definitely
>>>>>>>>>> should be added, but who knows when that's done... I mean, it's like
>>>>>>>>>> that for what, a decade? (: It's not even decided what it exactly
>>>>>>>>>> does, as there are many ways of approaching this. (I have my own
>>>>> idea
>>>>>>>>>> about what the right compromise would be, but others has other
>>>>>>>>>> ideas...)
>>>>>>>>>> 
>>>>>>>>>> Filtering lists bothers me because the template language should be
>>>>>>>>>> (and somewhat indeed is) specialized on listing things on fancy ways
>>>>>>>>>> that used to come up when generating document-like output. (If it
>>>>>>>>>> doesn't do things like that, you might as well use a general purpose
>>>>>>>>>> language.) Thus, that filter is unsolved (filtering with #if is
>>>>>>>>>> verbose and spoils #sep etc.) bothers me a lot.
>>>>>>>>>> 
>>>>>>>>>> BTW, ?filter and ?map is also especially handy in our case as
>>>>>>>>>> FreeMarker doesn't support building new sequences (sequences are
>>>>>>>>>> immutable). Although it has sequence concatenation with `+`, it's
>>>>> not
>>>>>>>>>> good for building a sequence one by one, unless the sequence will be
>>>>>>>>>> quite short.
>>>>>>>>>> 
>>>>>>>>> Good point.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Cheers,
>>>>>>>>>>> -- Denis.
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Thanks,
>>>>>>>>>> Daniel Dekany
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Thanks,
>>>>>>>> Daniel Dekany
>>>>>>>> 
>>>>>>>> 
>>>>>>> Thanks
>>>>>>> Christoph
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> --
>>>>> Thanks,
>>>>> Daniel Dekany
>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> Christoph Rüger, Geschäftsführer
>>>> Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
>>>> Programmieren - Automatisierung, Schnittstellen, Datenfeeds
>>>> 
>>>> Xing: https://www.xing.com/profile/Christoph_Rueger2
>>>> LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
>>>> 
>>> 
>>> --
>>> Thanks,
>>> Daniel Dekany
>>> 
>
>

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Denis Bredelet <br...@mac.com.INVALID>]

Hi Daniel,

> Wow, it's amazing!
> I'd spend some time testing it.
> 
> Thanks a lot!

Agreed, that looks wonderful. I want to test it.

Another use could be “combine” which applies a lambda to the list and one more list (zipping):

<#list products?combine(links, (p, l) -> linkify(p.name, l)) as clickableProduct>
...

Not sure I like the syntax I came up with. Maybe this would be better:

  products?combine(links)?map(p, l -> linkify(p.name, l))

Will that be useful though?

— Denis.

> Woonsan
> 
> On Sat, Feb 23, 2019 at 12:19 PM Daniel Dekany <dd...@apache.org> wrote:
>> 
>> I have pushed an implementation of the restricted lambdas we were
>> talking about. Guys, please review/test it.
>> 
>> I call these "local lambdas" in the source code (but I'm open for
>> suggestions), as the function they define can only be called in the
>> same variable scope where the lambda was (as we have no closures in
>> the template language, nor final variables). Also, they can only be
>> used as the parameters of the built-ins that explicitly support them.
>> As the subject shows, the main goal was just support filtering for
>> #list, without a nested #if, as that breaks #sep, #else, it?hasNext,
>> it?index, etc. (I will try to make more universal lambdas in FM3... I
>> guess it would be way too tricky in FM2.)
>> 
>> For now, I have only added two built-ins that support lambdas: ?filter
>> and ?map. Any ideas what other such built-ins would be often useful in
>> templates (with use case, if possible)?
>> 
>> Examples of using ?filter and ?map:
>> 
>>  <#list products?filter(it -> it.price < 1000) as product>
>>    ${product.name}
>>  </#list>
>> 
>>  <#list products?map(it -> it.name) as name>
>>    ${name}
>>  </#list>
>> 
>> Of course these built-ins aren't specific to #list, they can be used
>> anywhere. Naturally, they can be chained as well:
>> 
>>  <#assign chepProdNames = products
>>     ?filter(it -> it.price < 1000)
>>     ?map(it -> it.name)
>>> 
>> 
>> As a side note, ?filter and ?map also accepts FTL function-s and Java
>> methods as its parameter, not only lambdas.
>> 
>> A tricky aspect of this feature is lazy evaluation, in similar sense
>> as Java 8 Stream intermediate operations are lazy. On most places,
>> ?filer and ?map are eager, that is, they return a completed sequence
>> of items. That's because our restricted lambdas only work correctly
>> "locally". However, there are very common situations where it's clear
>> that we can use lazy ("streaming") evaluation, as we know where the
>> resulting stream of elements will be consumed:
>> 
>> - One such case is when these kind of built-ins are chained. Like in
>>  the last example, ?filter doesn't construct a List in memory,
>>  instead the elements just flow through it (some is dropped, as it's
>>  filtering), into ?map. Only the ?map at the end of the chain will
>>  built a List eagerly. Some other built-ins also allow the left-hand
>>  built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
>>  is built anywhere, instead the elements flow through both ?filter
>>  and ?map, and ?join just appends them to the StringBuilder where it
>>  creates its results.
>> 
>> - #list also enables lazy evaluation to its 1st parameter. So in the
>>  #list examples above, yet again no List is built in memory.
>> 
>> - There are other such cases, which I didn't implement yet. For
>>  example the sequence slice operator, like, xs?filter(f)[10..20],
>>  should allow lazy evaluation.
>> 
>> Feedback is welcome!
>> 
>> 
>> Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
>> 
>>> Hey Daniel,
>>> I'm very sorry, but I didn't make any progress with this. I think I was a
>>> bit over-motivated, but unfortunately I cannot spend more time on this for
>>> various reasons.
>>> You can take this over. I'm glad to help out with testing and feedback.
>>> 
>>> Thanks
>>> Christoph
>>> 
>>> 
>>> 
>>> 
>>> Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>>> :
>>> 
>>>> Any progress in this? I think I will give it a try in the coming days
>>>> otherwise.
>>>> 
>>>> 
>>>> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
>>>> 
>>>>> See my answers inline...
>>>>> 
>>>>> Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
>>>>> 
>>>>>> Thanks Daniel for your feedback. See my answers below
>>>>>> 
>>>>>> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
>>>> ddekany@apache.org
>>>>>>> :
>>>>>> 
>>>>>>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>>>>>>> 
>>>>>>>> Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
>>>>>>> ddekany@apache.org
>>>>>>>>> :
>>>>>>>> 
>>>>>>>>> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com>
>>>> a
>>>>>>>>> écrit :
>>>>>>>>>> 
>>>>>>>>>> Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>>>>>>>>> ddekany@apache.org
>>>>>>>>>> :
>>>>>>>>>> 
>>>>>>>>>> It's certainly tricky, but as far as I see possible (but then, who
>>>>>>>>>> 
>>>>>>>>>> knows what will one find when actually working on it). It's also a
>>>>>>>>>> feature missing a lot. It's especially missing for #list (I know
>>>> that
>>>>>>>>>> you need it for something else), because if you filter the items
>>>>>>>>>> inside #list with #if-s, then #sep, ?hasNext, etc. will not be
>>>> usable.
>>>>>>>>>> 
>>>>>>>>>> Let me say that I disagree here.
>>>>>>>>>> 
>>>>>>>>>> I do not think that closures are required for FreeMarker, nor that
>>>>>>> they
>>>>>>>>> are a good idea.
>>>>>>>>>> 
>>>>>>>>>> If we add new features to the FreeMarker *tempate engine* I would
>>>>>>>>>> rather we focus on multi-part macro body rather than an advanced
>>>>>>>>> language feature like closures.
>>>>>>>>>> 
>>>>>>>>>> You can add ?filter and ?map if you want, a simple expression as
>>>>>>>>> parameter should be enough.
>>>>>>>>> 
>>>>>>>>> Yes, as I said, we certainly start with only allowing lambdas in
>>>>>>>>> ?filter/?map, also certainly in ?contains.
>>>>>>>>> 
>>>>>>>> Would be enough in my opinion and very useful.
>>>>>>>> 
>>>>>>>> Is it possiblefor you to give some pointers to the code on how this
>>>> could
>>>>>>>> be implemented? I would maybe like to wrap my head around this a
>>>> little
>>>>>>> bit.
>>>>>>> 
>>>>>>> Please feel yourself encouraged! (:
>>>>>>> 
>>>>>>>> I started looking at seq_containsBI (
>>>>>>>> 
>>>>>>> 
>>>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
>>>>>>> )
>>>>>>>> and
>>>>>>>> and reverseBI (
>>>>>>>> 
>>>>>>> 
>>>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
>>>>>>> )
>>>>>>>> just to find something related (seq_containsBI checks something) and
>>>>>>>> reverseBI returns a new sequence.
>>>>>>>> What I haven't found is a function which takes an Expression as a
>>>>>>>> parameter.
>>>>>>>> Is there something similar already or would that be a new thing?
>>>>>>> 
>>>>>>> It's a new thing in that it will be part of the expression syntax
>>>>>>> (even if for now we will only allow lambdas as the parameters of a few
>>>>>>> built-ins, so that we can get away without closures). So it's a new
>>>>>>> Expression subclass, and has to be part of the parser (ftl.jj) as
>>>>>>> well.
>>>>>> 
>>>>>> Hmm, that parser stuff is new for me, it'll take me some time to get
>>>> into
>>>>>> it.
>>>>> 
>>>>> So it's a JavaCC lexer+parser. With a few twists... sorry, but this
>>>>> code has history... :)
>>>>> 
>>>>>>> As of lazy evaluation of parameters expressions, that's already
>>>>>>> done in the built-ins in BuiltInsWithParseTimeParameters, and you will
>>>>>>> see it's trivial to do, but the situation there is much simpler.
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> In principle, a LambdaExpression should evaluate to a
>>>>>>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
>>>>>>> the called built-in or whatever it is. But with the approach of
>>>>>>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
>>>>>>> just bind to the LambdaExpression directly, add a LocalContext that
>>>>>>> contains the lambda arguments, end evaluate the LambdaExpression right
>>>>>>> there, in the built-in implementation. Or at least at a very quick
>>>>>>> glance I think so.
>>>>>>> 
>>>>>> Not sure I can follow completely but that hint with*
>>>>>> BuiltInsWithParseTimeParameters* got me started, but at the moment I'm
>>>>>> stuck as I need to get more familiar with the internals of Freemarker.
>>>> I am
>>>>>> also not sure I am on the same page regarding the syntax we are aiming
>>>> for
>>>>>> and why I would need to extend the parser when there is something like
>>>>>> BuiltInsWithParseTimeParameters....
>>>>> 
>>>>> I assumed that the syntax will be similar to the Java lambda syntax
>>>>> (see below why), and that's of course needs lexer/parser changes.
>>>>> 
>>>>>> Here is an example what I have in mind:
>>>>>> I started with the ?filter() builtin. I had a syntax like this in mind:
>>>>>> 
>>>>>> *Example 1: ["a","b","c"]?filter(element, element == "c")*
>>>>>> *Example 2: ["a","b","c"]?filter(element, element == someOtherVariable,
>>>>>> someOtherVariable="c")*
>>>>> 
>>>>> Looking at the above one believe that the value of `element` is passed
>>>>> in as first argument, and the the value of `element == "c"` as the
>>>>> second, but that's not the case. It's much better if it's visible that
>>>>> you got some kind of anonymous function definition there without
>>>>> knowing about the "filter" built-in.
>>>>> 
>>>>> Java already has a syntax for expressing this kind of thing, so it
>>>>> would be better to use that familiar syntax. As far as I know it
>>>>> doesn't conflict with ours (it kind of it does, but not fatally).
>>>>> 
>>>>> Also, even if for now we only allow this in said built-ins, we
>>>>> shouldn't exclude the possibility of making this kind of expression
>>>>> accessible elsewhere as well. Then, on many places the parsed won't
>>>>> know what kind of value is expected (consider passing a lambda to an
>>>>> user defined directive for example), so a syntax like above won't
>>>>> work.
>>>>> 
>>>>>> Not sure if that's what you have in mind too, but to me it made sense
>>>> with
>>>>>> regards to BuiltInsWithParseTimeParameters and I could start without
>>>>>> touching parser stuff.
>>>>> 
>>>>> BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
>>>>> bit...), as a call to a such built-in looks like a call to any other
>>>>> built-in.
>>>>> 
>>>>>> *1st argument 'element'* would just be the iterator variable similar to
>>>>>> <#list ["a","b","c"] as *element*>
>>>>>> 2nd argument is the filter lambda expression... aka our filter condition
>>>>>> 3rd+n argument are optional parameters in case used in the lambda
>>>> expression
>>>>> 
>>>>> #list is special, similarly as `for ... in ...` is in most languages.
>>>>> It's not lambda-ish either; you can't write `as element/2` or such.
>>>>> 
>>>>>> So at first I was looking how <#list> works and found IteratorBlock, and
>>>>>> though I could reuse it somehow.
>>>>>> 
>>>>>> Here is some simple pseudo code I played around for the for Example 1:
>>>>>> 
>>>>>> static class filter_BI extends BuiltInWithParseTimeParameters {
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>        TemplateModel _eval(Environment env) throws TemplateException {
>>>>>> 
>>>>>>            // sequence
>>>>>> 
>>>>>>        TemplateModel targetValue = target.evalToNonMissing(env);
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>            List parameters = this.parameters;
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>            Expression iteratorAlias = (Expression) parameters.get(0);
>>>>>> 
>>>>>>            Expression conditionExpression = (Expression)
>>>> parameters.get(1);
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>            TemplateSequenceModel seq =  (TemplateSequenceModel)
>>>> target.eval
>>>>>> (env);
>>>>>> 
>>>>>>            for (int i = 0; i < seq.size(); i++) {
>>>>>> 
>>>>>>               TemplateModel cur = seq.get(i);
>>>>>> 
>>>>>> 
>>>>>>               // this is where I am stuck at the moment
>>>>>> 
>>>>>>               // I basically want to evaluate conditionExpression
>>>>>> where iteratorAlias
>>>>>> is basically what I passed as 'element'
>>>>>> 
>>>>>>               // I am not sure if or how LocalContext could come into
>>>> play
>>>>>> here
>>>>>> 
>>>>>>               // basically for each iteration I would assign the
>>>> current
>>>>>> loop element to a context variable with the name 'element'
>>>>>> 
>>>>>>               // and then evaluate conditionExpression with that
>>>> context.
>>>>>> 
>>>>>>               // if conditionExpression is "true" then I would populate
>>>>>> add the current sequence element 'cur'
>>>>>> 
>>>>>>               // to a new result-List.... and return that.... something
>>>>>> 
>>>>>>               // I wanted to reuse IteratorBlock here somehow, but
>>>> didn't
>>>>>> get it to work yet.
>>>>>> 
>>>>>>               // maybe this is a stupid idea, or we just need something
>>>>>> similar
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>            }
>>>>>> 
>>>>>> }
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Ok so far for my pseudo code....  Maybe you could give some more
>>>> pointers
>>>>>> based on that... in case this makes any sense ...
>>>>> 
>>>>> For LocalContext, Environment has a pushLocalContext method. See the
>>>>> calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
>>>>> Map, List).
>>>>> 
>>>>> To avoid running into more new things at once than necessary, perhaps
>>>>> you should start with seq?seq_contains(lambda). The end result should
>>>>> be like:
>>>>> 
>>>>>  users?contains(u -> u.paying)
>>>>> 
>>>>>>> Another similarity to BuiltInsWithParseTimeParameters is that we won't
>>>>>>> allow separating the `?someBI` and `(arg)`. Like, with the example of
>>>>>>> `cond?then(1, 2)`, you aren't allowed to do this:
>>>>>>> 
>>>>>>>  <#assign t=cond?then>
>>>>>>>  ${t(1, 2)}
>>>>>>> 
>>>>>>> That maybe looks natural, but most other built-ins allow that. Of
>>>>>>> course we can't allow that for ?filter etc., because then we need
>>>>>>> closures again.
>>>>>>> 
>>>>>>>>> Multi-part macro body is also planned. Means, I know it definitely
>>>>>>>>> should be added, but who knows when that's done... I mean, it's like
>>>>>>>>> that for what, a decade? (: It's not even decided what it exactly
>>>>>>>>> does, as there are many ways of approaching this. (I have my own
>>>> idea
>>>>>>>>> about what the right compromise would be, but others has other
>>>>>>>>> ideas...)
>>>>>>>>> 
>>>>>>>>> Filtering lists bothers me because the template language should be
>>>>>>>>> (and somewhat indeed is) specialized on listing things on fancy ways
>>>>>>>>> that used to come up when generating document-like output. (If it
>>>>>>>>> doesn't do things like that, you might as well use a general purpose
>>>>>>>>> language.) Thus, that filter is unsolved (filtering with #if is
>>>>>>>>> verbose and spoils #sep etc.) bothers me a lot.
>>>>>>>>> 
>>>>>>>>> BTW, ?filter and ?map is also especially handy in our case as
>>>>>>>>> FreeMarker doesn't support building new sequences (sequences are
>>>>>>>>> immutable). Although it has sequence concatenation with `+`, it's
>>>> not
>>>>>>>>> good for building a sequence one by one, unless the sequence will be
>>>>>>>>> quite short.
>>>>>>>>> 
>>>>>>>> Good point.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> -- Denis.
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Thanks,
>>>>>>>>> Daniel Dekany
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Daniel Dekany
>>>>>>> 
>>>>>>> 
>>>>>> Thanks
>>>>>> Christoph
>>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> Thanks,
>>>> Daniel Dekany
>>>> 
>>>> 
>>> 
>>> --
>>> Christoph Rüger, Geschäftsführer
>>> Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
>>> Programmieren - Automatisierung, Schnittstellen, Datenfeeds
>>> 
>>> Xing: https://www.xing.com/profile/Christoph_Rueger2
>>> LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
>>> 
>> 
>> --
>> Thanks,
>> Daniel Dekany
>> 

Re: Lambda Expressions - filter list without <#list> directive

[Woonsan Ko <wo...@apache.org>]

Wow, it's amazing!
I'd spend some time testing it.

Thanks a lot!

Woonsan

On Sat, Feb 23, 2019 at 12:19 PM Daniel Dekany <dd...@apache.org> wrote:
>
> I have pushed an implementation of the restricted lambdas we were
> talking about. Guys, please review/test it.
>
> I call these "local lambdas" in the source code (but I'm open for
> suggestions), as the function they define can only be called in the
> same variable scope where the lambda was (as we have no closures in
> the template language, nor final variables). Also, they can only be
> used as the parameters of the built-ins that explicitly support them.
> As the subject shows, the main goal was just support filtering for
> #list, without a nested #if, as that breaks #sep, #else, it?hasNext,
> it?index, etc. (I will try to make more universal lambdas in FM3... I
> guess it would be way too tricky in FM2.)
>
> For now, I have only added two built-ins that support lambdas: ?filter
> and ?map. Any ideas what other such built-ins would be often useful in
> templates (with use case, if possible)?
>
> Examples of using ?filter and ?map:
>
>   <#list products?filter(it -> it.price < 1000) as product>
>     ${product.name}
>   </#list>
>
>   <#list products?map(it -> it.name) as name>
>     ${name}
>   </#list>
>
> Of course these built-ins aren't specific to #list, they can be used
> anywhere. Naturally, they can be chained as well:
>
>   <#assign chepProdNames = products
>      ?filter(it -> it.price < 1000)
>      ?map(it -> it.name)
>   >
>
> As a side note, ?filter and ?map also accepts FTL function-s and Java
> methods as its parameter, not only lambdas.
>
> A tricky aspect of this feature is lazy evaluation, in similar sense
> as Java 8 Stream intermediate operations are lazy. On most places,
> ?filer and ?map are eager, that is, they return a completed sequence
> of items. That's because our restricted lambdas only work correctly
> "locally". However, there are very common situations where it's clear
> that we can use lazy ("streaming") evaluation, as we know where the
> resulting stream of elements will be consumed:
>
> - One such case is when these kind of built-ins are chained. Like in
>   the last example, ?filter doesn't construct a List in memory,
>   instead the elements just flow through it (some is dropped, as it's
>   filtering), into ?map. Only the ?map at the end of the chain will
>   built a List eagerly. Some other built-ins also allow the left-hand
>   built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
>   is built anywhere, instead the elements flow through both ?filter
>   and ?map, and ?join just appends them to the StringBuilder where it
>   creates its results.
>
> - #list also enables lazy evaluation to its 1st parameter. So in the
>   #list examples above, yet again no List is built in memory.
>
> - There are other such cases, which I didn't implement yet. For
>   example the sequence slice operator, like, xs?filter(f)[10..20],
>   should allow lazy evaluation.
>
> Feedback is welcome!
>
>
> Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
>
> > Hey Daniel,
> > I'm very sorry, but I didn't make any progress with this. I think I was a
> > bit over-motivated, but unfortunately I cannot spend more time on this for
> > various reasons.
> > You can take this over. I'm glad to help out with testing and feedback.
> >
> > Thanks
> > Christoph
> >
> >
> >
> >
> > Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <ddekany@apache.org
> >>:
> >
> >> Any progress in this? I think I will give it a try in the coming days
> >> otherwise.
> >>
> >>
> >> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
> >>
> >> > See my answers inline...
> >> >
> >> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
> >> >
> >> >> Thanks Daniel for your feedback. See my answers below
> >> >>
> >> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
> >> ddekany@apache.org
> >> >>>:
> >> >>
> >> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
> >> >>>
> >> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
> >> >>> ddekany@apache.org
> >> >>> >>:
> >> >>> >
> >> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
> >> >>> >>
> >> >>> >> > Hi,
> >> >>> >> >
> >> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com>
> >> a
> >> >>> >> écrit :
> >> >>> >> >
> >> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
> >> >>> >> ddekany@apache.org
> >> >>> >> > :
> >> >>> >> >
> >> >>> >> > It's certainly tricky, but as far as I see possible (but then, who
> >> >>> >> >
> >> >>> >> > knows what will one find when actually working on it). It's also a
> >> >>> >> > feature missing a lot. It's especially missing for #list (I know
> >> that
> >> >>> >> > you need it for something else), because if you filter the items
> >> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be
> >> usable.
> >> >>> >> >
> >> >>> >> > Let me say that I disagree here.
> >> >>> >> >
> >> >>> >> > I do not think that closures are required for FreeMarker, nor that
> >> >>> they
> >> >>> >> are a good idea.
> >> >>> >> >
> >> >>> >> > If we add new features to the FreeMarker *tempate engine* I would
> >> >>> >> > rather we focus on multi-part macro body rather than an advanced
> >> >>> >> language feature like closures.
> >> >>> >> >
> >> >>> >> > You can add ?filter and ?map if you want, a simple expression as
> >> >>> >> parameter should be enough.
> >> >>> >>
> >> >>> >> Yes, as I said, we certainly start with only allowing lambdas in
> >> >>> >> ?filter/?map, also certainly in ?contains.
> >> >>> >>
> >> >>> > Would be enough in my opinion and very useful.
> >> >>> >
> >> >>> > Is it possiblefor you to give some pointers to the code on how this
> >> could
> >> >>> > be implemented? I would maybe like to wrap my head around this a
> >> little
> >> >>> bit.
> >> >>>
> >> >>> Please feel yourself encouraged! (:
> >> >>>
> >> >>> > I started looking at seq_containsBI (
> >> >>> >
> >> >>>
> >> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
> >> >>> )
> >> >>> > and
> >> >>> > and reverseBI (
> >> >>> >
> >> >>>
> >> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
> >> >>> )
> >> >>> > just to find something related (seq_containsBI checks something) and
> >> >>> > reverseBI returns a new sequence.
> >> >>> > What I haven't found is a function which takes an Expression as a
> >> >>> > parameter.
> >> >>> > Is there something similar already or would that be a new thing?
> >> >>>
> >> >>> It's a new thing in that it will be part of the expression syntax
> >> >>> (even if for now we will only allow lambdas as the parameters of a few
> >> >>> built-ins, so that we can get away without closures). So it's a new
> >> >>> Expression subclass, and has to be part of the parser (ftl.jj) as
> >> >>> well.
> >> >>
> >> >> Hmm, that parser stuff is new for me, it'll take me some time to get
> >> into
> >> >> it.
> >> >
> >> > So it's a JavaCC lexer+parser. With a few twists... sorry, but this
> >> > code has history... :)
> >> >
> >> >>> As of lazy evaluation of parameters expressions, that's already
> >> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and you will
> >> >>> see it's trivial to do, but the situation there is much simpler.
> >> >>>
> >> >>
> >> >>
> >> >>>
> >> >>> In principle, a LambdaExpression should evaluate to a
> >> >>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
> >> >>> the called built-in or whatever it is. But with the approach of
> >> >>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
> >> >>> just bind to the LambdaExpression directly, add a LocalContext that
> >> >>> contains the lambda arguments, end evaluate the LambdaExpression right
> >> >>> there, in the built-in implementation. Or at least at a very quick
> >> >>> glance I think so.
> >> >>>
> >> >> Not sure I can follow completely but that hint with*
> >> >> BuiltInsWithParseTimeParameters* got me started, but at the moment I'm
> >> >> stuck as I need to get more familiar with the internals of Freemarker.
> >> I am
> >> >> also not sure I am on the same page regarding the syntax we are aiming
> >> for
> >> >> and why I would need to extend the parser when there is something like
> >> >> BuiltInsWithParseTimeParameters....
> >> >
> >> > I assumed that the syntax will be similar to the Java lambda syntax
> >> > (see below why), and that's of course needs lexer/parser changes.
> >> >
> >> >> Here is an example what I have in mind:
> >> >> I started with the ?filter() builtin. I had a syntax like this in mind:
> >> >>
> >> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
> >> >> *Example 2: ["a","b","c"]?filter(element, element == someOtherVariable,
> >> >> someOtherVariable="c")*
> >> >
> >> > Looking at the above one believe that the value of `element` is passed
> >> > in as first argument, and the the value of `element == "c"` as the
> >> > second, but that's not the case. It's much better if it's visible that
> >> > you got some kind of anonymous function definition there without
> >> > knowing about the "filter" built-in.
> >> >
> >> > Java already has a syntax for expressing this kind of thing, so it
> >> > would be better to use that familiar syntax. As far as I know it
> >> > doesn't conflict with ours (it kind of it does, but not fatally).
> >> >
> >> > Also, even if for now we only allow this in said built-ins, we
> >> > shouldn't exclude the possibility of making this kind of expression
> >> > accessible elsewhere as well. Then, on many places the parsed won't
> >> > know what kind of value is expected (consider passing a lambda to an
> >> > user defined directive for example), so a syntax like above won't
> >> > work.
> >> >
> >> >> Not sure if that's what you have in mind too, but to me it made sense
> >> with
> >> >> regards to BuiltInsWithParseTimeParameters and I could start without
> >> >> touching parser stuff.
> >> >
> >> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
> >> > bit...), as a call to a such built-in looks like a call to any other
> >> > built-in.
> >> >
> >> >> *1st argument 'element'* would just be the iterator variable similar to
> >> >> <#list ["a","b","c"] as *element*>
> >> >> 2nd argument is the filter lambda expression... aka our filter condition
> >> >> 3rd+n argument are optional parameters in case used in the lambda
> >> expression
> >> >
> >> > #list is special, similarly as `for ... in ...` is in most languages.
> >> > It's not lambda-ish either; you can't write `as element/2` or such.
> >> >
> >> >> So at first I was looking how <#list> works and found IteratorBlock, and
> >> >> though I could reuse it somehow.
> >> >>
> >> >> Here is some simple pseudo code I played around for the for Example 1:
> >> >>
> >> >> static class filter_BI extends BuiltInWithParseTimeParameters {
> >> >>
> >> >>
> >> >>
> >> >>         TemplateModel _eval(Environment env) throws TemplateException {
> >> >>
> >> >>             // sequence
> >> >>
> >> >>         TemplateModel targetValue = target.evalToNonMissing(env);
> >> >>
> >> >>
> >> >>
> >> >>             List parameters = this.parameters;
> >> >>
> >> >>
> >> >>
> >> >>             Expression iteratorAlias = (Expression) parameters.get(0);
> >> >>
> >> >>             Expression conditionExpression = (Expression)
> >> parameters.get(1);
> >> >>
> >> >>
> >> >>
> >> >>             TemplateSequenceModel seq =  (TemplateSequenceModel)
> >> target.eval
> >> >> (env);
> >> >>
> >> >>             for (int i = 0; i < seq.size(); i++) {
> >> >>
> >> >>                TemplateModel cur = seq.get(i);
> >> >>
> >> >>
> >> >>                // this is where I am stuck at the moment
> >> >>
> >> >>                // I basically want to evaluate conditionExpression
> >> >> where iteratorAlias
> >> >> is basically what I passed as 'element'
> >> >>
> >> >>                // I am not sure if or how LocalContext could come into
> >> play
> >> >> here
> >> >>
> >> >>                // basically for each iteration I would assign the
> >> current
> >> >> loop element to a context variable with the name 'element'
> >> >>
> >> >>                // and then evaluate conditionExpression with that
> >> context.
> >> >>
> >> >>                // if conditionExpression is "true" then I would populate
> >> >> add the current sequence element 'cur'
> >> >>
> >> >>                // to a new result-List.... and return that.... something
> >> >>
> >> >>                // I wanted to reuse IteratorBlock here somehow, but
> >> didn't
> >> >> get it to work yet.
> >> >>
> >> >>                // maybe this is a stupid idea, or we just need something
> >> >> similar
> >> >>
> >> >>
> >> >>
> >> >>             }
> >> >>
> >> >> }
> >> >>
> >> >>
> >> >>
> >> >> Ok so far for my pseudo code....  Maybe you could give some more
> >> pointers
> >> >> based on that... in case this makes any sense ...
> >> >
> >> > For LocalContext, Environment has a pushLocalContext method. See the
> >> > calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
> >> > Map, List).
> >> >
> >> > To avoid running into more new things at once than necessary, perhaps
> >> > you should start with seq?seq_contains(lambda). The end result should
> >> > be like:
> >> >
> >> >   users?contains(u -> u.paying)
> >> >
> >> >>> Another similarity to BuiltInsWithParseTimeParameters is that we won't
> >> >>> allow separating the `?someBI` and `(arg)`. Like, with the example of
> >> >>> `cond?then(1, 2)`, you aren't allowed to do this:
> >> >>>
> >> >>>   <#assign t=cond?then>
> >> >>>   ${t(1, 2)}
> >> >>>
> >> >>> That maybe looks natural, but most other built-ins allow that. Of
> >> >>> course we can't allow that for ?filter etc., because then we need
> >> >>> closures again.
> >> >>>
> >> >>> >> Multi-part macro body is also planned. Means, I know it definitely
> >> >>> >> should be added, but who knows when that's done... I mean, it's like
> >> >>> >> that for what, a decade? (: It's not even decided what it exactly
> >> >>> >> does, as there are many ways of approaching this. (I have my own
> >> idea
> >> >>> >> about what the right compromise would be, but others has other
> >> >>> >> ideas...)
> >> >>> >>
> >> >>> >> Filtering lists bothers me because the template language should be
> >> >>> >> (and somewhat indeed is) specialized on listing things on fancy ways
> >> >>> >> that used to come up when generating document-like output. (If it
> >> >>> >> doesn't do things like that, you might as well use a general purpose
> >> >>> >> language.) Thus, that filter is unsolved (filtering with #if is
> >> >>> >> verbose and spoils #sep etc.) bothers me a lot.
> >> >>> >>
> >> >>> >> BTW, ?filter and ?map is also especially handy in our case as
> >> >>> >> FreeMarker doesn't support building new sequences (sequences are
> >> >>> >> immutable). Although it has sequence concatenation with `+`, it's
> >> not
> >> >>> >> good for building a sequence one by one, unless the sequence will be
> >> >>> >> quite short.
> >> >>> >>
> >> >>> > Good point.
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >> > Cheers,
> >> >>> >> > -- Denis.
> >> >>> >>
> >> >>> >> --
> >> >>> >> Thanks,
> >> >>> >>  Daniel Dekany
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>>
> >> >>> --
> >> >>> Thanks,
> >> >>>  Daniel Dekany
> >> >>>
> >> >>>
> >> >> Thanks
> >> >> Christoph
> >> >>
> >> >
> >>
> >> --
> >> Thanks,
> >>  Daniel Dekany
> >>
> >>
> >
> > --
> > Christoph Rüger, Geschäftsführer
> > Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
> > Programmieren - Automatisierung, Schnittstellen, Datenfeeds
> >
> > Xing: https://www.xing.com/profile/Christoph_Rueger2
> > LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
> >
>
> --
> Thanks,
>  Daniel Dekany
>

Re: Lambda Expressions - filter list without <#list> directive

[D��niel D��k��ny <dd...@apache.org>]

I was thinking about this, and realized that there are two problems with "where" (though maybe "where" is still better than "filter" overall):

1. I bet users will often write products?where(price < 1000), instead of products?where(p -> p.price < 1000). In SQL, the context is implicitly the table row, so you don't have a lambda argument. With ?filter the same mistake is less likely, as your brain doesn't switch to SQL mode.

2. We also have ?map, ?take_while and ?drop_while. These are Java Stream API names. So after seeing these, may will think there must be ?filter as well. (Of course the error message can tell that it's "?where" instead, but it can be still annoying.)

For now I left it as ?filter, but it's still not too late to change it. Someone else has opinion?

On 2019/07/02 20:18:30, Denis Bredelet <br...@mac.com.INVALID> wrote: 
> 
> > Le 2 juil. 2019 à 20:29, Pete Helgren <pe...@valadd.com> a écrit :
> > 
> > As a more casual Java programmer, the "where" option is much clearer to me. I spend more time using FM syntax than changing the Java underneath, so from a "fading memory" standpoint, "where" would lead to fewer "What the....?" moments,  for me at least.
> 
> I prefer « where » for the reasons Daniel mentioned, also SQL uses WHERE.
> 
> I think SQL has as many users as Javascript, no?
> 
> — Denis.
> 
> > 
> > Pete Helgren
> > www.petesworkshop.com
> > GIAC Secure Software Programmer-Java
> > Twitter - Sys_i_Geek  IBM_i_Geek
> > 
> > On 7/2/2019 2:08 PM, Christoph Rüger wrote:
> >> Good point. Seems you are not the first ones stumbling on that one.
> >> I quickly searched around and found:
> >> 
> >> Similar question on SO:
> >> https://stackoverflow.com/questions/45939202/filter-naming-convention
> >> Javascript: filter :
> >> https://developer.mozilla.org/de/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
> >> Spark SQL -> "where" is an alias for "filter":
> >> https://stackoverflow.com/a/33887122/135535
> >> <https://stackoverflow.com/questions/33885979/difference-between-filter-and-where-in-scala-spark-sql>
> >> -> search for "filter" or "where" on
> >> https://spark.apache.org/docs/1.5.2/api/scala/index.html#org.apache.spark.sql.DataFrame
> >> R Statistics Language : filter
> >> https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html#filter-rows-with-filter
> >> 
> >> Python: filter https://www.geeksforgeeks.org/filter-in-python/
> >> Ruby: they use select:
> >> https://www.codementor.io/tips/8247613177/how-to-filter-arrays-of-data-in-ruby
> >> Kotlin: filter:
> >> https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/filter.html
> >> 
> >> This languages rank in the upper area of the Stackoverflow survey:
> >> https://insights.stackoverflow.com/survey/2019#technology-_-programming-scripting-and-markup-languages
> >> 
> >> I agree that "where" reads pretty nice. I like it. But "filter" seems to be
> >> found in multiple common languages supporting lambdaish syntax.
> >> Python and R is especially common in the data science / statistics
> >> community, which are different target group than e.g. Java-Programmers.
> >> Also web-developers these days are doing lots of javascript to build "html"
> >> websites / templates - and javascript also uses "filter".
> >> 
> >> My vote would still go for "filter", because I think we are working on
> >> lists of objects and objects are closer to "programming" than to "sql".
> >> Maybe the "where"-alias would be a compromise - but might also be confusing
> >> two have both.
> >> 
> >> What do others think?
> >> 
> >> Thanks
> >> Christoph
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> Am Di., 2. Juli 2019 um 20:27 Uhr schrieb Daniel Dekany <ddekany@apache.org
> >>> :
> >>> I wonder if "filter" is a good name. For Java 8 programmers it's
> >>> given, but otherwise I find it confusing, as it's not clear if you
> >>> specify what to filter out, or what to keep. Worse, I believe in every
> >>> day English "foo filter" or "filters foo" means removing foo-s because
> >>> you don't want them, which is just the opposite of the meaning in
> >>> Java. So I think "where", which is familiar for many from SQL (for
> >>> most Java programmers as well, but also for non-Java programmers),
> >>> would be better. Consider:
> >>> 
> >>>   users?filter(user -> user.inactive)
> >>> 
> >>> VS
> >>> 
> >>>   users?where(user -> user.inactive)
> >>> 
> >>> The first can be easily misunderstood as removing the inactive users,
> >>> while the meaning of the second is obvious.
> >>> 
> 
> 

Re: Lambda Expressions - filter list without <#list> directive

[Denis Bredelet <br...@mac.com.INVALID>]

> Le 2 juil. 2019 à 20:29, Pete Helgren <pe...@valadd.com> a écrit :
> 
> As a more casual Java programmer, the "where" option is much clearer to me. I spend more time using FM syntax than changing the Java underneath, so from a "fading memory" standpoint, "where" would lead to fewer "What the....?" moments,  for me at least.

I prefer « where » for the reasons Daniel mentioned, also SQL uses WHERE.

I think SQL has as many users as Javascript, no?

— Denis.

> 
> Pete Helgren
> www.petesworkshop.com
> GIAC Secure Software Programmer-Java
> Twitter - Sys_i_Geek  IBM_i_Geek
> 
> On 7/2/2019 2:08 PM, Christoph Rüger wrote:
>> Good point. Seems you are not the first ones stumbling on that one.
>> I quickly searched around and found:
>> 
>> Similar question on SO:
>> https://stackoverflow.com/questions/45939202/filter-naming-convention
>> Javascript: filter :
>> https://developer.mozilla.org/de/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
>> Spark SQL -> "where" is an alias for "filter":
>> https://stackoverflow.com/a/33887122/135535
>> <https://stackoverflow.com/questions/33885979/difference-between-filter-and-where-in-scala-spark-sql>
>> -> search for "filter" or "where" on
>> https://spark.apache.org/docs/1.5.2/api/scala/index.html#org.apache.spark.sql.DataFrame
>> R Statistics Language : filter
>> https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html#filter-rows-with-filter
>> 
>> Python: filter https://www.geeksforgeeks.org/filter-in-python/
>> Ruby: they use select:
>> https://www.codementor.io/tips/8247613177/how-to-filter-arrays-of-data-in-ruby
>> Kotlin: filter:
>> https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/filter.html
>> 
>> This languages rank in the upper area of the Stackoverflow survey:
>> https://insights.stackoverflow.com/survey/2019#technology-_-programming-scripting-and-markup-languages
>> 
>> I agree that "where" reads pretty nice. I like it. But "filter" seems to be
>> found in multiple common languages supporting lambdaish syntax.
>> Python and R is especially common in the data science / statistics
>> community, which are different target group than e.g. Java-Programmers.
>> Also web-developers these days are doing lots of javascript to build "html"
>> websites / templates - and javascript also uses "filter".
>> 
>> My vote would still go for "filter", because I think we are working on
>> lists of objects and objects are closer to "programming" than to "sql".
>> Maybe the "where"-alias would be a compromise - but might also be confusing
>> two have both.
>> 
>> What do others think?
>> 
>> Thanks
>> Christoph
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Am Di., 2. Juli 2019 um 20:27 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>> :
>>> I wonder if "filter" is a good name. For Java 8 programmers it's
>>> given, but otherwise I find it confusing, as it's not clear if you
>>> specify what to filter out, or what to keep. Worse, I believe in every
>>> day English "foo filter" or "filters foo" means removing foo-s because
>>> you don't want them, which is just the opposite of the meaning in
>>> Java. So I think "where", which is familiar for many from SQL (for
>>> most Java programmers as well, but also for non-Java programmers),
>>> would be better. Consider:
>>> 
>>>   users?filter(user -> user.inactive)
>>> 
>>> VS
>>> 
>>>   users?where(user -> user.inactive)
>>> 
>>> The first can be easily misunderstood as removing the inactive users,
>>> while the meaning of the second is obvious.
>>> 

Re: Lambda Expressions - filter list without <#list> directive

[Pete Helgren <pe...@valadd.com>]

As a more casual Java programmer, the "where" option is much clearer to 
me. I spend more time using FM syntax than changing the Java underneath, 
so from a "fading memory" standpoint, "where" would lead to fewer "What 
the....?" moments,  for me at least.

Pete Helgren
www.petesworkshop.com
GIAC Secure Software Programmer-Java
Twitter - Sys_i_Geek  IBM_i_Geek

On 7/2/2019 2:08 PM, Christoph Rüger wrote:
> Good point. Seems you are not the first ones stumbling on that one.
> I quickly searched around and found:
>
> Similar question on SO:
> https://stackoverflow.com/questions/45939202/filter-naming-convention
> Javascript: filter :
> https://developer.mozilla.org/de/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
> Spark SQL -> "where" is an alias for "filter":
> https://stackoverflow.com/a/33887122/135535
> <https://stackoverflow.com/questions/33885979/difference-between-filter-and-where-in-scala-spark-sql>
> -> search for "filter" or "where" on
> https://spark.apache.org/docs/1.5.2/api/scala/index.html#org.apache.spark.sql.DataFrame
> R Statistics Language : filter
> https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html#filter-rows-with-filter
>
> Python: filter https://www.geeksforgeeks.org/filter-in-python/
> Ruby: they use select:
> https://www.codementor.io/tips/8247613177/how-to-filter-arrays-of-data-in-ruby
> Kotlin: filter:
> https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/filter.html
>
> This languages rank in the upper area of the Stackoverflow survey:
> https://insights.stackoverflow.com/survey/2019#technology-_-programming-scripting-and-markup-languages
>
> I agree that "where" reads pretty nice. I like it. But "filter" seems to be
> found in multiple common languages supporting lambdaish syntax.
> Python and R is especially common in the data science / statistics
> community, which are different target group than e.g. Java-Programmers.
> Also web-developers these days are doing lots of javascript to build "html"
> websites / templates - and javascript also uses "filter".
>
> My vote would still go for "filter", because I think we are working on
> lists of objects and objects are closer to "programming" than to "sql".
> Maybe the "where"-alias would be a compromise - but might also be confusing
> two have both.
>
> What do others think?
>
> Thanks
> Christoph
>
>
>
>
>
>
>
> Am Di., 2. Juli 2019 um 20:27 Uhr schrieb Daniel Dekany <ddekany@apache.org
>> :
>> I wonder if "filter" is a good name. For Java 8 programmers it's
>> given, but otherwise I find it confusing, as it's not clear if you
>> specify what to filter out, or what to keep. Worse, I believe in every
>> day English "foo filter" or "filters foo" means removing foo-s because
>> you don't want them, which is just the opposite of the meaning in
>> Java. So I think "where", which is familiar for many from SQL (for
>> most Java programmers as well, but also for non-Java programmers),
>> would be better. Consider:
>>
>>    users?filter(user -> user.inactive)
>>
>> VS
>>
>>    users?where(user -> user.inactive)
>>
>> The first can be easily misunderstood as removing the inactive users,
>> while the meaning of the second is obvious.
>>
>>
>> Tuesday, July 2, 2019, 2:57:52 PM, Christoph Rüger wrote:
>>
>>> Thanks for the heads up. Very nice. We will run our test suite to see if
>>> those test are still green.
>>>
>>> Am Mo., 1. Juli 2019 um 09:30 Uhr schrieb Daniel Dekany <
>> ddekany@freemail.hu
>>>> :
>>>> Since then I have also made a change that ensures that if the lambda
>>>> argument is null (which in FTL is the same as if the variable isn't
>>>> there at all), then it will not fall back to find an identically named
>>>> variable in higher variable scopes. This is important when doing
>>>> things like:
>>>>
>>>>    <#-- filters out null-s -->
>>>>    myList?filter(it -> it??)
>>>>
>>>> because if some day someone adds a variable called "it" to the
>>>> data-model, then suddenly the above won't filter out the null-s.
>>>>
>>>> The same thing was always an issue with #list loop variables as well,
>>>> also with #nested arguments. So I have added a configuration setting
>>>> called "fallbackOnNullLoopVariable", which is by default true
>>>> (unfortunate historical baggage... but we can't break backward
>>>> compatibility). If you set it to false, then this will print "N/A" at
>>>> null list items, rather than "Clashing variable in higher scope":
>>>>
>>>> <#assign it = "Clashing variable in higher scope">
>>>> <#list myList as it>
>>>>    ${it!'N/A'}
>>>> </#list>
>>>>
>>>> These changes are pushed and deployed to the Apache snapshot Maven
>>>> repo in both branches.
>>>>
>>>>
>>>> So, apart from documentation, the local lambda feature is about ready,
>>>> or so I hope. I'm worried of rough edges though, so I think I will add
>>>> lambda support to some more builtins (?seq_contains, ?sort_by), and
>>>> explore some more use cases... If you have your own that you actually
>>>> keep running into, or want to be in the 2.3.29, tell it.
>>>>
>>>>
>>>> Monday, June 24, 2019, 1:59:21 AM, Daniel Dekany wrote:
>>>>
>>>>> Well, I'm not exactly fast nowadays either... Anyway, I have pushed
>>>>> and deployed to the snapshot repo the changes I was talking about
>>>>> recently. That is, ?map or ?filter won't make a sequence out of an
>>>>> enumerable non-sequence (typically an Iterator) anymore. Because, it
>>>>> was the concern that if hugeResultSet is an Iterator because it's
>>>>> huge, then someone might writes:
>>>>>
>>>>>    <#assign transformed = hugeResultSet?map(it -> something(it))>
>>>>>    <#list transformed as it>
>>>>>
>>>>> instead of just
>>>>>
>>>>>    <#list hugeResultSet?map(it -> something(it)) as it>
>>>>>
>>>>> and thus consuming a lot of memory without realizing it. So now if
>>>>> hugeResultSet wasn't already a sequence (List-like), the assignment
>>>>> will be an error, since we can't safely store a lazily transformed
>>>>> collection (lambdas will break), and we can't condense it down to a
>>>>> sequence (List-like thing) automatically either, as that might
>>>>> consumes too much memory. If hugeResultSet was a sequence, then it's
>>>>> not an error, as we assume that keeping all of it in memory is fine,
>>>>> as the original was stored there as well (in practice, most of the
>>>>> times... in principle we can't know).
>>>>>
>>>>> Now if the user feels confident about it, they can still write:
>>>>>
>>>>>    <#assign transformed = hugeResultSet?map(it ->
>> something(it))?sequence>
>>>>> Similarly, hugeResultSet?map(it -> something(it))[index] will be an
>>>>> error, as [index] is for sequences only, and ?map will not change a
>>>>> non-sequence to a sequence anymore. Similarly, if the user feels
>>>>> confident about it, they can write hugeResultSet?map(it ->
>>>>> something(it))?sequence[index].
>>>>>
>>>>> An interesting consequence of these is that ?sequence is now a bit
>>>>> smarter than before. Like if you write myIterator?sequnce[n], it will
>>>>> not fetch the elements into an in-memory sequence, it just skips n
>>>>> elements from myIterators, and returns the nth one. Similarly,
>>>>> myIterator?sequence?size won't store the elements in memory, it just
>>>>> counts them.
>>>>>
>>>>> As an interesting note, these two are also identically efficient:
>>>>>
>>>>>    <#assign seq = hugeResultSet?filter(it -> something(it))?sequence>
>>>>>    <#assign seq = hugeResultSet?sequence?filter(it -> something(it))>
>>>>>
>>>>> In both cases the actual conversion to a sequence (in-memory list)
>>>>> happens only just before assigning the value to seq. Once again,
>>>>> ?sequence now just means "it's OK to treat this as a sequence, however
>>>>> inefficient it is", and not "convert it to sequence right now".
>>>>>
>>>>>
>>>>> Friday, June 7, 2019, 10:38:50 AM, Christoph Rüger wrote:
>>>>>
>>>>>> These optimisations sound great. I will try to run some tests within
>> the
>>>>>> next weeks. A bit busy lately.
>>>>>> Thanks
>>>>>> Christoph
>>>>>>
>>>>>> Am Mi., 29. Mai 2019 um 23:55 Uhr schrieb Daniel Dekany <
>>>> ddekany@apache.org
>>>>>>> :
>>>>>>> Tuesday, April 2, 2019, 12:10:16 PM, Christoph Rüger wrote:
>>>>>>>
>>>>>>> [snip]
>>>>>>>>> Well, if you fear users jumping on ?filter/?map outside #list
>> for no
>>>>>>>>> good enough reason, there can be some option to handle that. But
>> I
>>>>>>>>> don't think restricting the usage to #list is a good compromise
>> as
>>>> the
>>>>>>>>> default.
>>>>>>>> I agree. Just keep as it is.
>>>>>>>>
>>>>>>>>>>> I'm not sure how efficiently could a configuration setting
>> catch
>>>>>>> these
>>>>>>>>>>> cases, or if it should be addressed on that level.
>>>>>>>>>> Maybe let's postpone configurability discussion a bit until the
>>>> above
>>>>>>> is
>>>>>>>>>> more clear.
>>>>>>>>> In the light of the above, I think we can start thinking about
>> that
>>>>>>>>> now.
>>>>>>>> On that note on configurability: Would it be possible to
>>>> programmatically
>>>>>>>> influence the Collection (Sequence) which is created under the
>> hood?
>>>>>>>> E.g. by specifying a Factory? I ask because we are using something
>>>> like
>>>>>>>> this (
>>>>>>>>
>> https://dzone.com/articles/a-filebasedcollection-in-java-for-big-collections
>>>>>>> )
>>>>>>>> in other places for large collections. I know it is very specific,
>>>> but
>>>>>>> just
>>>>>>>> wanted to bring it up.
>>>>>>> [snip]
>>>>>>>
>>>>>>> I think a good approach would be to ban the *implicit* collection of
>>>>>>> the result, when the filtered/mapped source is an Iterator, or other
>>>>>>> similar stream-like object that's often used for enumerating a huge
>>>>>>> number of elements. So for example, let's say you have this:
>>>>>>>
>>>>>>>    <#assign xs2 = xs?filter(f)>
>>>>>>>
>>>>>>> If xs is List-like, then this will work. Since the xs List fits into
>>>>>>> the memory (although a List can be backed by disk, that's rather
>>>>>>> rare), hopefully it's not the kind of data amount that can't fit
>> into
>>>>>>> the memory again (as xs2). On the other hand, if xs is an
>>>>>>> Iterator-like object, then the above statement fails, with the hint
>>>>>>> that xs?filter(f)?sequence would work, but might consumes a lot of
>>>>>>> memory.
>>>>>>>
>>>>>>> This is also consistent with how xs[i] works in the existing
>>>>>>> FreeMarker versions. That only works if xs is List-like (an FTL
>>>>>>> sequence). While xs[i] would be trivial to implement even if xs is
>>>>>>> Iterator-like, we don't do that as it's not efficient for a high i,
>>>>>>> and so the template author is probably not meant to do that. If he
>>>>>>> knows what's he doing though, he can write xs?sequence[i]. Yes,
>> that's
>>>>>>> very inefficient if you only use [] once on that sequence, but you
>> see
>>>>>>> the logic. map/filter breaks it, as xs?filter(f)[i] works even if xs
>>>>>>> is an Iterator, because filter/map currently always returns a
>>>>>>> sequence. If xs is Iteartor-like, then I want filter/map to return
>> an
>>>>>>> Iterator-like as well, so then [] will fail on it.
>>>>>>>
>>>>>>> As a side note, I will make ?sequence smarter too, so that
>>>>>>> xs?sequence[i] won't actually build a sequence if xs is
>> Iterator-like.
>>>>>>> It just have to skip the first i elements after all. (The ?sequence
>> is
>>>>>>> still required there. It basically says: "I know what I'm doing,
>> treat
>>>>>>> this as a sequence.")
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>>   Daniel Dekany
>>>>>>>
>>>>>>>
>>>> --
>>>> Thanks,
>>>>   Daniel Dekany
>>>>
>>>>
>>> --
>>> Christoph Rüger, Geschäftsführer
>>> Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
>>> Programmieren - Automatisierung, Schnittstellen, Datenfeeds
>>>
>>> Xing: https://www.xing.com/profile/Christoph_Rueger2
>>> LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
>>>
>> --
>> Thanks,
>>   Daniel Dekany
>>
>>

Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

Good point. Seems you are not the first ones stumbling on that one.
I quickly searched around and found:

Similar question on SO:
https://stackoverflow.com/questions/45939202/filter-naming-convention
Javascript: filter :
https://developer.mozilla.org/de/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
Spark SQL -> "where" is an alias for "filter":
https://stackoverflow.com/a/33887122/135535
<https://stackoverflow.com/questions/33885979/difference-between-filter-and-where-in-scala-spark-sql>
-> search for "filter" or "where" on
https://spark.apache.org/docs/1.5.2/api/scala/index.html#org.apache.spark.sql.DataFrame
R Statistics Language : filter
https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html#filter-rows-with-filter

Python: filter https://www.geeksforgeeks.org/filter-in-python/
Ruby: they use select:
https://www.codementor.io/tips/8247613177/how-to-filter-arrays-of-data-in-ruby
Kotlin: filter:
https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/filter.html

This languages rank in the upper area of the Stackoverflow survey:
https://insights.stackoverflow.com/survey/2019#technology-_-programming-scripting-and-markup-languages

I agree that "where" reads pretty nice. I like it. But "filter" seems to be
found in multiple common languages supporting lambdaish syntax.
Python and R is especially common in the data science / statistics
community, which are different target group than e.g. Java-Programmers.
Also web-developers these days are doing lots of javascript to build "html"
websites / templates - and javascript also uses "filter".

My vote would still go for "filter", because I think we are working on
lists of objects and objects are closer to "programming" than to "sql".
Maybe the "where"-alias would be a compromise - but might also be confusing
two have both.

What do others think?

Thanks
Christoph







Am Di., 2. Juli 2019 um 20:27 Uhr schrieb Daniel Dekany <ddekany@apache.org
>:

> I wonder if "filter" is a good name. For Java 8 programmers it's
> given, but otherwise I find it confusing, as it's not clear if you
> specify what to filter out, or what to keep. Worse, I believe in every
> day English "foo filter" or "filters foo" means removing foo-s because
> you don't want them, which is just the opposite of the meaning in
> Java. So I think "where", which is familiar for many from SQL (for
> most Java programmers as well, but also for non-Java programmers),
> would be better. Consider:
>
>   users?filter(user -> user.inactive)
>
> VS
>
>   users?where(user -> user.inactive)
>
> The first can be easily misunderstood as removing the inactive users,
> while the meaning of the second is obvious.
>
>
> Tuesday, July 2, 2019, 2:57:52 PM, Christoph Rüger wrote:
>
> > Thanks for the heads up. Very nice. We will run our test suite to see if
> > those test are still green.
> >
> > Am Mo., 1. Juli 2019 um 09:30 Uhr schrieb Daniel Dekany <
> ddekany@freemail.hu
> >>:
> >
> >> Since then I have also made a change that ensures that if the lambda
> >> argument is null (which in FTL is the same as if the variable isn't
> >> there at all), then it will not fall back to find an identically named
> >> variable in higher variable scopes. This is important when doing
> >> things like:
> >>
> >>   <#-- filters out null-s -->
> >>   myList?filter(it -> it??)
> >>
> >> because if some day someone adds a variable called "it" to the
> >> data-model, then suddenly the above won't filter out the null-s.
> >>
> >> The same thing was always an issue with #list loop variables as well,
> >> also with #nested arguments. So I have added a configuration setting
> >> called "fallbackOnNullLoopVariable", which is by default true
> >> (unfortunate historical baggage... but we can't break backward
> >> compatibility). If you set it to false, then this will print "N/A" at
> >> null list items, rather than "Clashing variable in higher scope":
> >>
> >> <#assign it = "Clashing variable in higher scope">
> >> <#list myList as it>
> >>   ${it!'N/A'}
> >> </#list>
> >>
> >> These changes are pushed and deployed to the Apache snapshot Maven
> >> repo in both branches.
> >>
> >>
> >> So, apart from documentation, the local lambda feature is about ready,
> >> or so I hope. I'm worried of rough edges though, so I think I will add
> >> lambda support to some more builtins (?seq_contains, ?sort_by), and
> >> explore some more use cases... If you have your own that you actually
> >> keep running into, or want to be in the 2.3.29, tell it.
> >>
> >>
> >> Monday, June 24, 2019, 1:59:21 AM, Daniel Dekany wrote:
> >>
> >> > Well, I'm not exactly fast nowadays either... Anyway, I have pushed
> >> > and deployed to the snapshot repo the changes I was talking about
> >> > recently. That is, ?map or ?filter won't make a sequence out of an
> >> > enumerable non-sequence (typically an Iterator) anymore. Because, it
> >> > was the concern that if hugeResultSet is an Iterator because it's
> >> > huge, then someone might writes:
> >> >
> >> >   <#assign transformed = hugeResultSet?map(it -> something(it))>
> >> >   <#list transformed as it>
> >> >
> >> > instead of just
> >> >
> >> >   <#list hugeResultSet?map(it -> something(it)) as it>
> >> >
> >> > and thus consuming a lot of memory without realizing it. So now if
> >> > hugeResultSet wasn't already a sequence (List-like), the assignment
> >> > will be an error, since we can't safely store a lazily transformed
> >> > collection (lambdas will break), and we can't condense it down to a
> >> > sequence (List-like thing) automatically either, as that might
> >> > consumes too much memory. If hugeResultSet was a sequence, then it's
> >> > not an error, as we assume that keeping all of it in memory is fine,
> >> > as the original was stored there as well (in practice, most of the
> >> > times... in principle we can't know).
> >> >
> >> > Now if the user feels confident about it, they can still write:
> >> >
> >> >   <#assign transformed = hugeResultSet?map(it ->
> something(it))?sequence>
> >> >
> >> > Similarly, hugeResultSet?map(it -> something(it))[index] will be an
> >> > error, as [index] is for sequences only, and ?map will not change a
> >> > non-sequence to a sequence anymore. Similarly, if the user feels
> >> > confident about it, they can write hugeResultSet?map(it ->
> >> > something(it))?sequence[index].
> >> >
> >> > An interesting consequence of these is that ?sequence is now a bit
> >> > smarter than before. Like if you write myIterator?sequnce[n], it will
> >> > not fetch the elements into an in-memory sequence, it just skips n
> >> > elements from myIterators, and returns the nth one. Similarly,
> >> > myIterator?sequence?size won't store the elements in memory, it just
> >> > counts them.
> >> >
> >> > As an interesting note, these two are also identically efficient:
> >> >
> >> >   <#assign seq = hugeResultSet?filter(it -> something(it))?sequence>
> >> >   <#assign seq = hugeResultSet?sequence?filter(it -> something(it))>
> >> >
> >> > In both cases the actual conversion to a sequence (in-memory list)
> >> > happens only just before assigning the value to seq. Once again,
> >> > ?sequence now just means "it's OK to treat this as a sequence, however
> >> > inefficient it is", and not "convert it to sequence right now".
> >> >
> >> >
> >> > Friday, June 7, 2019, 10:38:50 AM, Christoph Rüger wrote:
> >> >
> >> >> These optimisations sound great. I will try to run some tests within
> the
> >> >> next weeks. A bit busy lately.
> >> >> Thanks
> >> >> Christoph
> >> >>
> >> >> Am Mi., 29. Mai 2019 um 23:55 Uhr schrieb Daniel Dekany <
> >> ddekany@apache.org
> >> >>>:
> >> >>
> >> >>> Tuesday, April 2, 2019, 12:10:16 PM, Christoph Rüger wrote:
> >> >>>
> >> >>> [snip]
> >> >>> >> Well, if you fear users jumping on ?filter/?map outside #list
> for no
> >> >>> >> good enough reason, there can be some option to handle that. But
> I
> >> >>> >> don't think restricting the usage to #list is a good compromise
> as
> >> the
> >> >>> >> default.
> >> >>> >
> >> >>> > I agree. Just keep as it is.
> >> >>> >
> >> >>> >> >> I'm not sure how efficiently could a configuration setting
> catch
> >> >>> these
> >> >>> >> >> cases, or if it should be addressed on that level.
> >> >>> >> >
> >> >>> >> > Maybe let's postpone configurability discussion a bit until the
> >> above
> >> >>> is
> >> >>> >> > more clear.
> >> >>> >>
> >> >>> >> In the light of the above, I think we can start thinking about
> that
> >> >>> >> now.
> >> >>> >
> >> >>> > On that note on configurability: Would it be possible to
> >> programmatically
> >> >>> > influence the Collection (Sequence) which is created under the
> hood?
> >> >>> > E.g. by specifying a Factory? I ask because we are using something
> >> like
> >> >>> > this (
> >> >>> >
> >> >>>
> >>
> https://dzone.com/articles/a-filebasedcollection-in-java-for-big-collections
> >> >>> )
> >> >>> > in other places for large collections. I know it is very specific,
> >> but
> >> >>> just
> >> >>> > wanted to bring it up.
> >> >>> [snip]
> >> >>>
> >> >>> I think a good approach would be to ban the *implicit* collection of
> >> >>> the result, when the filtered/mapped source is an Iterator, or other
> >> >>> similar stream-like object that's often used for enumerating a huge
> >> >>> number of elements. So for example, let's say you have this:
> >> >>>
> >> >>>   <#assign xs2 = xs?filter(f)>
> >> >>>
> >> >>> If xs is List-like, then this will work. Since the xs List fits into
> >> >>> the memory (although a List can be backed by disk, that's rather
> >> >>> rare), hopefully it's not the kind of data amount that can't fit
> into
> >> >>> the memory again (as xs2). On the other hand, if xs is an
> >> >>> Iterator-like object, then the above statement fails, with the hint
> >> >>> that xs?filter(f)?sequence would work, but might consumes a lot of
> >> >>> memory.
> >> >>>
> >> >>> This is also consistent with how xs[i] works in the existing
> >> >>> FreeMarker versions. That only works if xs is List-like (an FTL
> >> >>> sequence). While xs[i] would be trivial to implement even if xs is
> >> >>> Iterator-like, we don't do that as it's not efficient for a high i,
> >> >>> and so the template author is probably not meant to do that. If he
> >> >>> knows what's he doing though, he can write xs?sequence[i]. Yes,
> that's
> >> >>> very inefficient if you only use [] once on that sequence, but you
> see
> >> >>> the logic. map/filter breaks it, as xs?filter(f)[i] works even if xs
> >> >>> is an Iterator, because filter/map currently always returns a
> >> >>> sequence. If xs is Iteartor-like, then I want filter/map to return
> an
> >> >>> Iterator-like as well, so then [] will fail on it.
> >> >>>
> >> >>> As a side note, I will make ?sequence smarter too, so that
> >> >>> xs?sequence[i] won't actually build a sequence if xs is
> Iterator-like.
> >> >>> It just have to skip the first i elements after all. (The ?sequence
> is
> >> >>> still required there. It basically says: "I know what I'm doing,
> treat
> >> >>> this as a sequence.")
> >> >>>
> >> >>> --
> >> >>> Thanks,
> >> >>>  Daniel Dekany
> >> >>>
> >> >>>
> >> >>
> >> >
> >>
> >> --
> >> Thanks,
> >>  Daniel Dekany
> >>
> >>
> >
> > --
> > Christoph Rüger, Geschäftsführer
> > Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
> > Programmieren - Automatisierung, Schnittstellen, Datenfeeds
> >
> > Xing: https://www.xing.com/profile/Christoph_Rueger2
> > LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
> >
>
> --
> Thanks,
>  Daniel Dekany
>
>

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 
5596493Internet: https://synesty.com <https://synesty.com>
Informationen 
zum Datenschutz: https://synesty.com/datenschutz 
<https://synesty.com/datenschutz>

Geschäftsführer: Christoph Rüger

Unternehmenssitz: Jena
Handelsregister B beim Amtsgericht: Jena

Handelsregister-Nummer: HRB 508766
Ust-IdNr.: DE287564982

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

I wonder if "filter" is a good name. For Java 8 programmers it's
given, but otherwise I find it confusing, as it's not clear if you
specify what to filter out, or what to keep. Worse, I believe in every
day English "foo filter" or "filters foo" means removing foo-s because
you don't want them, which is just the opposite of the meaning in
Java. So I think "where", which is familiar for many from SQL (for
most Java programmers as well, but also for non-Java programmers),
would be better. Consider:

  users?filter(user -> user.inactive)

VS

  users?where(user -> user.inactive)

The first can be easily misunderstood as removing the inactive users,
while the meaning of the second is obvious.


Tuesday, July 2, 2019, 2:57:52 PM, Christoph Rüger wrote:

> Thanks for the heads up. Very nice. We will run our test suite to see if
> those test are still green.
>
> Am Mo., 1. Juli 2019 um 09:30 Uhr schrieb Daniel Dekany <ddekany@freemail.hu
>>:
>
>> Since then I have also made a change that ensures that if the lambda
>> argument is null (which in FTL is the same as if the variable isn't
>> there at all), then it will not fall back to find an identically named
>> variable in higher variable scopes. This is important when doing
>> things like:
>>
>>   <#-- filters out null-s -->
>>   myList?filter(it -> it??)
>>
>> because if some day someone adds a variable called "it" to the
>> data-model, then suddenly the above won't filter out the null-s.
>>
>> The same thing was always an issue with #list loop variables as well,
>> also with #nested arguments. So I have added a configuration setting
>> called "fallbackOnNullLoopVariable", which is by default true
>> (unfortunate historical baggage... but we can't break backward
>> compatibility). If you set it to false, then this will print "N/A" at
>> null list items, rather than "Clashing variable in higher scope":
>>
>> <#assign it = "Clashing variable in higher scope">
>> <#list myList as it>
>>   ${it!'N/A'}
>> </#list>
>>
>> These changes are pushed and deployed to the Apache snapshot Maven
>> repo in both branches.
>>
>>
>> So, apart from documentation, the local lambda feature is about ready,
>> or so I hope. I'm worried of rough edges though, so I think I will add
>> lambda support to some more builtins (?seq_contains, ?sort_by), and
>> explore some more use cases... If you have your own that you actually
>> keep running into, or want to be in the 2.3.29, tell it.
>>
>>
>> Monday, June 24, 2019, 1:59:21 AM, Daniel Dekany wrote:
>>
>> > Well, I'm not exactly fast nowadays either... Anyway, I have pushed
>> > and deployed to the snapshot repo the changes I was talking about
>> > recently. That is, ?map or ?filter won't make a sequence out of an
>> > enumerable non-sequence (typically an Iterator) anymore. Because, it
>> > was the concern that if hugeResultSet is an Iterator because it's
>> > huge, then someone might writes:
>> >
>> >   <#assign transformed = hugeResultSet?map(it -> something(it))>
>> >   <#list transformed as it>
>> >
>> > instead of just
>> >
>> >   <#list hugeResultSet?map(it -> something(it)) as it>
>> >
>> > and thus consuming a lot of memory without realizing it. So now if
>> > hugeResultSet wasn't already a sequence (List-like), the assignment
>> > will be an error, since we can't safely store a lazily transformed
>> > collection (lambdas will break), and we can't condense it down to a
>> > sequence (List-like thing) automatically either, as that might
>> > consumes too much memory. If hugeResultSet was a sequence, then it's
>> > not an error, as we assume that keeping all of it in memory is fine,
>> > as the original was stored there as well (in practice, most of the
>> > times... in principle we can't know).
>> >
>> > Now if the user feels confident about it, they can still write:
>> >
>> >   <#assign transformed = hugeResultSet?map(it -> something(it))?sequence>
>> >
>> > Similarly, hugeResultSet?map(it -> something(it))[index] will be an
>> > error, as [index] is for sequences only, and ?map will not change a
>> > non-sequence to a sequence anymore. Similarly, if the user feels
>> > confident about it, they can write hugeResultSet?map(it ->
>> > something(it))?sequence[index].
>> >
>> > An interesting consequence of these is that ?sequence is now a bit
>> > smarter than before. Like if you write myIterator?sequnce[n], it will
>> > not fetch the elements into an in-memory sequence, it just skips n
>> > elements from myIterators, and returns the nth one. Similarly,
>> > myIterator?sequence?size won't store the elements in memory, it just
>> > counts them.
>> >
>> > As an interesting note, these two are also identically efficient:
>> >
>> >   <#assign seq = hugeResultSet?filter(it -> something(it))?sequence>
>> >   <#assign seq = hugeResultSet?sequence?filter(it -> something(it))>
>> >
>> > In both cases the actual conversion to a sequence (in-memory list)
>> > happens only just before assigning the value to seq. Once again,
>> > ?sequence now just means "it's OK to treat this as a sequence, however
>> > inefficient it is", and not "convert it to sequence right now".
>> >
>> >
>> > Friday, June 7, 2019, 10:38:50 AM, Christoph Rüger wrote:
>> >
>> >> These optimisations sound great. I will try to run some tests within the
>> >> next weeks. A bit busy lately.
>> >> Thanks
>> >> Christoph
>> >>
>> >> Am Mi., 29. Mai 2019 um 23:55 Uhr schrieb Daniel Dekany <
>> ddekany@apache.org
>> >>>:
>> >>
>> >>> Tuesday, April 2, 2019, 12:10:16 PM, Christoph Rüger wrote:
>> >>>
>> >>> [snip]
>> >>> >> Well, if you fear users jumping on ?filter/?map outside #list for no
>> >>> >> good enough reason, there can be some option to handle that. But I
>> >>> >> don't think restricting the usage to #list is a good compromise as
>> the
>> >>> >> default.
>> >>> >
>> >>> > I agree. Just keep as it is.
>> >>> >
>> >>> >> >> I'm not sure how efficiently could a configuration setting catch
>> >>> these
>> >>> >> >> cases, or if it should be addressed on that level.
>> >>> >> >
>> >>> >> > Maybe let's postpone configurability discussion a bit until the
>> above
>> >>> is
>> >>> >> > more clear.
>> >>> >>
>> >>> >> In the light of the above, I think we can start thinking about that
>> >>> >> now.
>> >>> >
>> >>> > On that note on configurability: Would it be possible to
>> programmatically
>> >>> > influence the Collection (Sequence) which is created under the hood?
>> >>> > E.g. by specifying a Factory? I ask because we are using something
>> like
>> >>> > this (
>> >>> >
>> >>>
>> https://dzone.com/articles/a-filebasedcollection-in-java-for-big-collections
>> >>> )
>> >>> > in other places for large collections. I know it is very specific,
>> but
>> >>> just
>> >>> > wanted to bring it up.
>> >>> [snip]
>> >>>
>> >>> I think a good approach would be to ban the *implicit* collection of
>> >>> the result, when the filtered/mapped source is an Iterator, or other
>> >>> similar stream-like object that's often used for enumerating a huge
>> >>> number of elements. So for example, let's say you have this:
>> >>>
>> >>>   <#assign xs2 = xs?filter(f)>
>> >>>
>> >>> If xs is List-like, then this will work. Since the xs List fits into
>> >>> the memory (although a List can be backed by disk, that's rather
>> >>> rare), hopefully it's not the kind of data amount that can't fit into
>> >>> the memory again (as xs2). On the other hand, if xs is an
>> >>> Iterator-like object, then the above statement fails, with the hint
>> >>> that xs?filter(f)?sequence would work, but might consumes a lot of
>> >>> memory.
>> >>>
>> >>> This is also consistent with how xs[i] works in the existing
>> >>> FreeMarker versions. That only works if xs is List-like (an FTL
>> >>> sequence). While xs[i] would be trivial to implement even if xs is
>> >>> Iterator-like, we don't do that as it's not efficient for a high i,
>> >>> and so the template author is probably not meant to do that. If he
>> >>> knows what's he doing though, he can write xs?sequence[i]. Yes, that's
>> >>> very inefficient if you only use [] once on that sequence, but you see
>> >>> the logic. map/filter breaks it, as xs?filter(f)[i] works even if xs
>> >>> is an Iterator, because filter/map currently always returns a
>> >>> sequence. If xs is Iteartor-like, then I want filter/map to return an
>> >>> Iterator-like as well, so then [] will fail on it.
>> >>>
>> >>> As a side note, I will make ?sequence smarter too, so that
>> >>> xs?sequence[i] won't actually build a sequence if xs is Iterator-like.
>> >>> It just have to skip the first i elements after all. (The ?sequence is
>> >>> still required there. It basically says: "I know what I'm doing, treat
>> >>> this as a sequence.")
>> >>>
>> >>> --
>> >>> Thanks,
>> >>>  Daniel Dekany
>> >>>
>> >>>
>> >>
>> >
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
>
> -- 
> Christoph Rüger, Geschäftsführer
> Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
> Programmieren - Automatisierung, Schnittstellen, Datenfeeds
>
> Xing: https://www.xing.com/profile/Christoph_Rueger2
> LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
>

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

Thanks for the heads up. Very nice. We will run our test suite to see if
those test are still green.

Am Mo., 1. Juli 2019 um 09:30 Uhr schrieb Daniel Dekany <ddekany@freemail.hu
>:

> Since then I have also made a change that ensures that if the lambda
> argument is null (which in FTL is the same as if the variable isn't
> there at all), then it will not fall back to find an identically named
> variable in higher variable scopes. This is important when doing
> things like:
>
>   <#-- filters out null-s -->
>   myList?filter(it -> it??)
>
> because if some day someone adds a variable called "it" to the
> data-model, then suddenly the above won't filter out the null-s.
>
> The same thing was always an issue with #list loop variables as well,
> also with #nested arguments. So I have added a configuration setting
> called "fallbackOnNullLoopVariable", which is by default true
> (unfortunate historical baggage... but we can't break backward
> compatibility). If you set it to false, then this will print "N/A" at
> null list items, rather than "Clashing variable in higher scope":
>
> <#assign it = "Clashing variable in higher scope">
> <#list myList as it>
>   ${it!'N/A'}
> </#list>
>
> These changes are pushed and deployed to the Apache snapshot Maven
> repo in both branches.
>
>
> So, apart from documentation, the local lambda feature is about ready,
> or so I hope. I'm worried of rough edges though, so I think I will add
> lambda support to some more builtins (?seq_contains, ?sort_by), and
> explore some more use cases... If you have your own that you actually
> keep running into, or want to be in the 2.3.29, tell it.
>
>
> Monday, June 24, 2019, 1:59:21 AM, Daniel Dekany wrote:
>
> > Well, I'm not exactly fast nowadays either... Anyway, I have pushed
> > and deployed to the snapshot repo the changes I was talking about
> > recently. That is, ?map or ?filter won't make a sequence out of an
> > enumerable non-sequence (typically an Iterator) anymore. Because, it
> > was the concern that if hugeResultSet is an Iterator because it's
> > huge, then someone might writes:
> >
> >   <#assign transformed = hugeResultSet?map(it -> something(it))>
> >   <#list transformed as it>
> >
> > instead of just
> >
> >   <#list hugeResultSet?map(it -> something(it)) as it>
> >
> > and thus consuming a lot of memory without realizing it. So now if
> > hugeResultSet wasn't already a sequence (List-like), the assignment
> > will be an error, since we can't safely store a lazily transformed
> > collection (lambdas will break), and we can't condense it down to a
> > sequence (List-like thing) automatically either, as that might
> > consumes too much memory. If hugeResultSet was a sequence, then it's
> > not an error, as we assume that keeping all of it in memory is fine,
> > as the original was stored there as well (in practice, most of the
> > times... in principle we can't know).
> >
> > Now if the user feels confident about it, they can still write:
> >
> >   <#assign transformed = hugeResultSet?map(it -> something(it))?sequence>
> >
> > Similarly, hugeResultSet?map(it -> something(it))[index] will be an
> > error, as [index] is for sequences only, and ?map will not change a
> > non-sequence to a sequence anymore. Similarly, if the user feels
> > confident about it, they can write hugeResultSet?map(it ->
> > something(it))?sequence[index].
> >
> > An interesting consequence of these is that ?sequence is now a bit
> > smarter than before. Like if you write myIterator?sequnce[n], it will
> > not fetch the elements into an in-memory sequence, it just skips n
> > elements from myIterators, and returns the nth one. Similarly,
> > myIterator?sequence?size won't store the elements in memory, it just
> > counts them.
> >
> > As an interesting note, these two are also identically efficient:
> >
> >   <#assign seq = hugeResultSet?filter(it -> something(it))?sequence>
> >   <#assign seq = hugeResultSet?sequence?filter(it -> something(it))>
> >
> > In both cases the actual conversion to a sequence (in-memory list)
> > happens only just before assigning the value to seq. Once again,
> > ?sequence now just means "it's OK to treat this as a sequence, however
> > inefficient it is", and not "convert it to sequence right now".
> >
> >
> > Friday, June 7, 2019, 10:38:50 AM, Christoph Rüger wrote:
> >
> >> These optimisations sound great. I will try to run some tests within the
> >> next weeks. A bit busy lately.
> >> Thanks
> >> Christoph
> >>
> >> Am Mi., 29. Mai 2019 um 23:55 Uhr schrieb Daniel Dekany <
> ddekany@apache.org
> >>>:
> >>
> >>> Tuesday, April 2, 2019, 12:10:16 PM, Christoph Rüger wrote:
> >>>
> >>> [snip]
> >>> >> Well, if you fear users jumping on ?filter/?map outside #list for no
> >>> >> good enough reason, there can be some option to handle that. But I
> >>> >> don't think restricting the usage to #list is a good compromise as
> the
> >>> >> default.
> >>> >
> >>> > I agree. Just keep as it is.
> >>> >
> >>> >> >> I'm not sure how efficiently could a configuration setting catch
> >>> these
> >>> >> >> cases, or if it should be addressed on that level.
> >>> >> >
> >>> >> > Maybe let's postpone configurability discussion a bit until the
> above
> >>> is
> >>> >> > more clear.
> >>> >>
> >>> >> In the light of the above, I think we can start thinking about that
> >>> >> now.
> >>> >
> >>> > On that note on configurability: Would it be possible to
> programmatically
> >>> > influence the Collection (Sequence) which is created under the hood?
> >>> > E.g. by specifying a Factory? I ask because we are using something
> like
> >>> > this (
> >>> >
> >>>
> https://dzone.com/articles/a-filebasedcollection-in-java-for-big-collections
> >>> )
> >>> > in other places for large collections. I know it is very specific,
> but
> >>> just
> >>> > wanted to bring it up.
> >>> [snip]
> >>>
> >>> I think a good approach would be to ban the *implicit* collection of
> >>> the result, when the filtered/mapped source is an Iterator, or other
> >>> similar stream-like object that's often used for enumerating a huge
> >>> number of elements. So for example, let's say you have this:
> >>>
> >>>   <#assign xs2 = xs?filter(f)>
> >>>
> >>> If xs is List-like, then this will work. Since the xs List fits into
> >>> the memory (although a List can be backed by disk, that's rather
> >>> rare), hopefully it's not the kind of data amount that can't fit into
> >>> the memory again (as xs2). On the other hand, if xs is an
> >>> Iterator-like object, then the above statement fails, with the hint
> >>> that xs?filter(f)?sequence would work, but might consumes a lot of
> >>> memory.
> >>>
> >>> This is also consistent with how xs[i] works in the existing
> >>> FreeMarker versions. That only works if xs is List-like (an FTL
> >>> sequence). While xs[i] would be trivial to implement even if xs is
> >>> Iterator-like, we don't do that as it's not efficient for a high i,
> >>> and so the template author is probably not meant to do that. If he
> >>> knows what's he doing though, he can write xs?sequence[i]. Yes, that's
> >>> very inefficient if you only use [] once on that sequence, but you see
> >>> the logic. map/filter breaks it, as xs?filter(f)[i] works even if xs
> >>> is an Iterator, because filter/map currently always returns a
> >>> sequence. If xs is Iteartor-like, then I want filter/map to return an
> >>> Iterator-like as well, so then [] will fail on it.
> >>>
> >>> As a side note, I will make ?sequence smarter too, so that
> >>> xs?sequence[i] won't actually build a sequence if xs is Iterator-like.
> >>> It just have to skip the first i elements after all. (The ?sequence is
> >>> still required there. It basically says: "I know what I'm doing, treat
> >>> this as a sequence.")
> >>>
> >>> --
> >>> Thanks,
> >>>  Daniel Dekany
> >>>
> >>>
> >>
> >
>
> --
> Thanks,
>  Daniel Dekany
>
>

-- 
Christoph Rüger, Geschäftsführer
Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
Programmieren - Automatisierung, Schnittstellen, Datenfeeds

Xing: https://www.xing.com/profile/Christoph_Rueger2
LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 
5596493Internet: https://synesty.com <https://synesty.com>
Informationen 
zum Datenschutz: https://synesty.com/datenschutz 
<https://synesty.com/datenschutz>

Geschäftsführer: Christoph Rüger

Unternehmenssitz: Jena
Handelsregister B beim Amtsgericht: Jena

Handelsregister-Nummer: HRB 508766
Ust-IdNr.: DE287564982

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@freemail.hu>]

Since then I have also made a change that ensures that if the lambda
argument is null (which in FTL is the same as if the variable isn't
there at all), then it will not fall back to find an identically named
variable in higher variable scopes. This is important when doing
things like:

  <#-- filters out null-s -->
  myList?filter(it -> it??)

because if some day someone adds a variable called "it" to the
data-model, then suddenly the above won't filter out the null-s.

The same thing was always an issue with #list loop variables as well,
also with #nested arguments. So I have added a configuration setting
called "fallbackOnNullLoopVariable", which is by default true
(unfortunate historical baggage... but we can't break backward
compatibility). If you set it to false, then this will print "N/A" at
null list items, rather than "Clashing variable in higher scope":

<#assign it = "Clashing variable in higher scope">
<#list myList as it>
  ${it!'N/A'}
</#list>

These changes are pushed and deployed to the Apache snapshot Maven
repo in both branches.


So, apart from documentation, the local lambda feature is about ready,
or so I hope. I'm worried of rough edges though, so I think I will add
lambda support to some more builtins (?seq_contains, ?sort_by), and
explore some more use cases... If you have your own that you actually
keep running into, or want to be in the 2.3.29, tell it.


Monday, June 24, 2019, 1:59:21 AM, Daniel Dekany wrote:

> Well, I'm not exactly fast nowadays either... Anyway, I have pushed
> and deployed to the snapshot repo the changes I was talking about
> recently. That is, ?map or ?filter won't make a sequence out of an
> enumerable non-sequence (typically an Iterator) anymore. Because, it
> was the concern that if hugeResultSet is an Iterator because it's
> huge, then someone might writes:
>
>   <#assign transformed = hugeResultSet?map(it -> something(it))>
>   <#list transformed as it>
>
> instead of just
>
>   <#list hugeResultSet?map(it -> something(it)) as it>
>
> and thus consuming a lot of memory without realizing it. So now if
> hugeResultSet wasn't already a sequence (List-like), the assignment
> will be an error, since we can't safely store a lazily transformed
> collection (lambdas will break), and we can't condense it down to a
> sequence (List-like thing) automatically either, as that might
> consumes too much memory. If hugeResultSet was a sequence, then it's
> not an error, as we assume that keeping all of it in memory is fine,
> as the original was stored there as well (in practice, most of the
> times... in principle we can't know).
>
> Now if the user feels confident about it, they can still write:
>
>   <#assign transformed = hugeResultSet?map(it -> something(it))?sequence>
>
> Similarly, hugeResultSet?map(it -> something(it))[index] will be an
> error, as [index] is for sequences only, and ?map will not change a
> non-sequence to a sequence anymore. Similarly, if the user feels
> confident about it, they can write hugeResultSet?map(it ->
> something(it))?sequence[index].
>
> An interesting consequence of these is that ?sequence is now a bit
> smarter than before. Like if you write myIterator?sequnce[n], it will
> not fetch the elements into an in-memory sequence, it just skips n
> elements from myIterators, and returns the nth one. Similarly,
> myIterator?sequence?size won't store the elements in memory, it just
> counts them.
>
> As an interesting note, these two are also identically efficient:
>
>   <#assign seq = hugeResultSet?filter(it -> something(it))?sequence>
>   <#assign seq = hugeResultSet?sequence?filter(it -> something(it))>
>
> In both cases the actual conversion to a sequence (in-memory list)
> happens only just before assigning the value to seq. Once again,
> ?sequence now just means "it's OK to treat this as a sequence, however
> inefficient it is", and not "convert it to sequence right now".
>
>
> Friday, June 7, 2019, 10:38:50 AM, Christoph Rüger wrote:
>
>> These optimisations sound great. I will try to run some tests within the
>> next weeks. A bit busy lately.
>> Thanks
>> Christoph
>>
>> Am Mi., 29. Mai 2019 um 23:55 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>>:
>>
>>> Tuesday, April 2, 2019, 12:10:16 PM, Christoph Rüger wrote:
>>>
>>> [snip]
>>> >> Well, if you fear users jumping on ?filter/?map outside #list for no
>>> >> good enough reason, there can be some option to handle that. But I
>>> >> don't think restricting the usage to #list is a good compromise as the
>>> >> default.
>>> >
>>> > I agree. Just keep as it is.
>>> >
>>> >> >> I'm not sure how efficiently could a configuration setting catch
>>> these
>>> >> >> cases, or if it should be addressed on that level.
>>> >> >
>>> >> > Maybe let's postpone configurability discussion a bit until the above
>>> is
>>> >> > more clear.
>>> >>
>>> >> In the light of the above, I think we can start thinking about that
>>> >> now.
>>> >
>>> > On that note on configurability: Would it be possible to programmatically
>>> > influence the Collection (Sequence) which is created under the hood?
>>> > E.g. by specifying a Factory? I ask because we are using something like
>>> > this (
>>> >
>>> https://dzone.com/articles/a-filebasedcollection-in-java-for-big-collections
>>> )
>>> > in other places for large collections. I know it is very specific, but
>>> just
>>> > wanted to bring it up.
>>> [snip]
>>>
>>> I think a good approach would be to ban the *implicit* collection of
>>> the result, when the filtered/mapped source is an Iterator, or other
>>> similar stream-like object that's often used for enumerating a huge
>>> number of elements. So for example, let's say you have this:
>>>
>>>   <#assign xs2 = xs?filter(f)>
>>>
>>> If xs is List-like, then this will work. Since the xs List fits into
>>> the memory (although a List can be backed by disk, that's rather
>>> rare), hopefully it's not the kind of data amount that can't fit into
>>> the memory again (as xs2). On the other hand, if xs is an
>>> Iterator-like object, then the above statement fails, with the hint
>>> that xs?filter(f)?sequence would work, but might consumes a lot of
>>> memory.
>>>
>>> This is also consistent with how xs[i] works in the existing
>>> FreeMarker versions. That only works if xs is List-like (an FTL
>>> sequence). While xs[i] would be trivial to implement even if xs is
>>> Iterator-like, we don't do that as it's not efficient for a high i,
>>> and so the template author is probably not meant to do that. If he
>>> knows what's he doing though, he can write xs?sequence[i]. Yes, that's
>>> very inefficient if you only use [] once on that sequence, but you see
>>> the logic. map/filter breaks it, as xs?filter(f)[i] works even if xs
>>> is an Iterator, because filter/map currently always returns a
>>> sequence. If xs is Iteartor-like, then I want filter/map to return an
>>> Iterator-like as well, so then [] will fail on it.
>>>
>>> As a side note, I will make ?sequence smarter too, so that
>>> xs?sequence[i] won't actually build a sequence if xs is Iterator-like.
>>> It just have to skip the first i elements after all. (The ?sequence is
>>> still required there. It basically says: "I know what I'm doing, treat
>>> this as a sequence.")
>>>
>>> --
>>> Thanks,
>>>  Daniel Dekany
>>>
>>>
>>
>

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

Well, I'm not exactly fast nowadays either... Anyway, I have pushed
and deployed to the snapshot repo the changes I was talking about
recently. That is, ?map or ?filter won't make a sequence out of an
enumerable non-sequence (typically an Iterator) anymore. Because, it
was the concern that if hugeResultSet is an Iterator because it's
huge, then someone might writes:

  <#assign transformed = hugeResultSet?map(it -> something(it))>
  <#list transformed as it>

instead of just

  <#list hugeResultSet?map(it -> something(it)) as it>

and thus consuming a lot of memory without realizing it. So now if
hugeResultSet wasn't already a sequence (List-like), the assignment
will be an error, since we can't safely store a lazily transformed
collection (lambdas will break), and we can't condense it down to a
sequence (List-like thing) automatically either, as that might
consumes too much memory. If hugeResultSet was a sequence, then it's
not an error, as we assume that keeping all of it in memory is fine,
as the original was stored there as well (in practice, most of the
times... in principle we can't know).

Now if the user feels confident about it, they can still write:

  <#assign transformed = hugeResultSet?map(it -> something(it))?sequence>

Similarly, hugeResultSet?map(it -> something(it))[index] will be an
error, as [index] is for sequences only, and ?map will not change a
non-sequence to a sequence anymore. Similarly, if the user feels
confident about it, they can write hugeResultSet?map(it ->
something(it))?sequence[index].

An interesting consequence of these is that ?sequence is now a bit
smarter than before. Like if you write myIterator?sequnce[n], it will
not fetch the elements into an in-memory sequence, it just skips n
elements from myIterators, and returns the nth one. Similarly,
myIterator?sequence?size won't store the elements in memory, it just
counts them.

As an interesting note, these two are also identically efficient:

  <#assign seq = hugeResultSet?filter(it -> something(it))?sequence>
  <#assign seq = hugeResultSet?sequence?filter(it -> something(it))>

In both cases the actual conversion to a sequence (in-memory list)
happens only just before assigning the value to seq. Once again,
?sequence now just means "it's OK to treat this as a sequence, however
inefficient it is", and not "convert it to sequence right now".


Friday, June 7, 2019, 10:38:50 AM, Christoph Rüger wrote:

> These optimisations sound great. I will try to run some tests within the
> next weeks. A bit busy lately.
> Thanks
> Christoph
>
> Am Mi., 29. Mai 2019 um 23:55 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>:
>
>> Tuesday, April 2, 2019, 12:10:16 PM, Christoph Rüger wrote:
>>
>> [snip]
>> >> Well, if you fear users jumping on ?filter/?map outside #list for no
>> >> good enough reason, there can be some option to handle that. But I
>> >> don't think restricting the usage to #list is a good compromise as the
>> >> default.
>> >
>> > I agree. Just keep as it is.
>> >
>> >> >> I'm not sure how efficiently could a configuration setting catch
>> these
>> >> >> cases, or if it should be addressed on that level.
>> >> >
>> >> > Maybe let's postpone configurability discussion a bit until the above
>> is
>> >> > more clear.
>> >>
>> >> In the light of the above, I think we can start thinking about that
>> >> now.
>> >
>> > On that note on configurability: Would it be possible to programmatically
>> > influence the Collection (Sequence) which is created under the hood?
>> > E.g. by specifying a Factory? I ask because we are using something like
>> > this (
>> >
>> https://dzone.com/articles/a-filebasedcollection-in-java-for-big-collections
>> )
>> > in other places for large collections. I know it is very specific, but
>> just
>> > wanted to bring it up.
>> [snip]
>>
>> I think a good approach would be to ban the *implicit* collection of
>> the result, when the filtered/mapped source is an Iterator, or other
>> similar stream-like object that's often used for enumerating a huge
>> number of elements. So for example, let's say you have this:
>>
>>   <#assign xs2 = xs?filter(f)>
>>
>> If xs is List-like, then this will work. Since the xs List fits into
>> the memory (although a List can be backed by disk, that's rather
>> rare), hopefully it's not the kind of data amount that can't fit into
>> the memory again (as xs2). On the other hand, if xs is an
>> Iterator-like object, then the above statement fails, with the hint
>> that xs?filter(f)?sequence would work, but might consumes a lot of
>> memory.
>>
>> This is also consistent with how xs[i] works in the existing
>> FreeMarker versions. That only works if xs is List-like (an FTL
>> sequence). While xs[i] would be trivial to implement even if xs is
>> Iterator-like, we don't do that as it's not efficient for a high i,
>> and so the template author is probably not meant to do that. If he
>> knows what's he doing though, he can write xs?sequence[i]. Yes, that's
>> very inefficient if you only use [] once on that sequence, but you see
>> the logic. map/filter breaks it, as xs?filter(f)[i] works even if xs
>> is an Iterator, because filter/map currently always returns a
>> sequence. If xs is Iteartor-like, then I want filter/map to return an
>> Iterator-like as well, so then [] will fail on it.
>>
>> As a side note, I will make ?sequence smarter too, so that
>> xs?sequence[i] won't actually build a sequence if xs is Iterator-like.
>> It just have to skip the first i elements after all. (The ?sequence is
>> still required there. It basically says: "I know what I'm doing, treat
>> this as a sequence.")
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
>

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

These optimisations sound great. I will try to run some tests within the
next weeks. A bit busy lately.
Thanks
Christoph

Am Mi., 29. Mai 2019 um 23:55 Uhr schrieb Daniel Dekany <ddekany@apache.org
>:

> Tuesday, April 2, 2019, 12:10:16 PM, Christoph Rüger wrote:
>
> [snip]
> >> Well, if you fear users jumping on ?filter/?map outside #list for no
> >> good enough reason, there can be some option to handle that. But I
> >> don't think restricting the usage to #list is a good compromise as the
> >> default.
> >
> > I agree. Just keep as it is.
> >
> >> >> I'm not sure how efficiently could a configuration setting catch
> these
> >> >> cases, or if it should be addressed on that level.
> >> >
> >> > Maybe let's postpone configurability discussion a bit until the above
> is
> >> > more clear.
> >>
> >> In the light of the above, I think we can start thinking about that
> >> now.
> >
> > On that note on configurability: Would it be possible to programmatically
> > influence the Collection (Sequence) which is created under the hood?
> > E.g. by specifying a Factory? I ask because we are using something like
> > this (
> >
> https://dzone.com/articles/a-filebasedcollection-in-java-for-big-collections
> )
> > in other places for large collections. I know it is very specific, but
> just
> > wanted to bring it up.
> [snip]
>
> I think a good approach would be to ban the *implicit* collection of
> the result, when the filtered/mapped source is an Iterator, or other
> similar stream-like object that's often used for enumerating a huge
> number of elements. So for example, let's say you have this:
>
>   <#assign xs2 = xs?filter(f)>
>
> If xs is List-like, then this will work. Since the xs List fits into
> the memory (although a List can be backed by disk, that's rather
> rare), hopefully it's not the kind of data amount that can't fit into
> the memory again (as xs2). On the other hand, if xs is an
> Iterator-like object, then the above statement fails, with the hint
> that xs?filter(f)?sequence would work, but might consumes a lot of
> memory.
>
> This is also consistent with how xs[i] works in the existing
> FreeMarker versions. That only works if xs is List-like (an FTL
> sequence). While xs[i] would be trivial to implement even if xs is
> Iterator-like, we don't do that as it's not efficient for a high i,
> and so the template author is probably not meant to do that. If he
> knows what's he doing though, he can write xs?sequence[i]. Yes, that's
> very inefficient if you only use [] once on that sequence, but you see
> the logic. map/filter breaks it, as xs?filter(f)[i] works even if xs
> is an Iterator, because filter/map currently always returns a
> sequence. If xs is Iteartor-like, then I want filter/map to return an
> Iterator-like as well, so then [] will fail on it.
>
> As a side note, I will make ?sequence smarter too, so that
> xs?sequence[i] won't actually build a sequence if xs is Iterator-like.
> It just have to skip the first i elements after all. (The ?sequence is
> still required there. It basically says: "I know what I'm doing, treat
> this as a sequence.")
>
> --
> Thanks,
>  Daniel Dekany
>
>

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 
5596493Internet: https://synesty.com <https://synesty.com>
Informationen 
zum Datenschutz: https://synesty.com/datenschutz 
<https://synesty.com/datenschutz>

Geschäftsführer: Christoph Rüger

Unternehmenssitz: Jena
Handelsregister B beim Amtsgericht: Jena

Handelsregister-Nummer: HRB 508766
Ust-IdNr.: DE287564982

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

Tuesday, April 2, 2019, 12:10:16 PM, Christoph Rüger wrote:

[snip]
>> Well, if you fear users jumping on ?filter/?map outside #list for no
>> good enough reason, there can be some option to handle that. But I
>> don't think restricting the usage to #list is a good compromise as the
>> default.
>
> I agree. Just keep as it is.
>
>> >> I'm not sure how efficiently could a configuration setting catch these
>> >> cases, or if it should be addressed on that level.
>> >
>> > Maybe let's postpone configurability discussion a bit until the above is
>> > more clear.
>>
>> In the light of the above, I think we can start thinking about that
>> now.
>
> On that note on configurability: Would it be possible to programmatically
> influence the Collection (Sequence) which is created under the hood?
> E.g. by specifying a Factory? I ask because we are using something like
> this (
> https://dzone.com/articles/a-filebasedcollection-in-java-for-big-collections)
> in other places for large collections. I know it is very specific, but just
> wanted to bring it up.
[snip]

I think a good approach would be to ban the *implicit* collection of
the result, when the filtered/mapped source is an Iterator, or other
similar stream-like object that's often used for enumerating a huge
number of elements. So for example, let's say you have this:

  <#assign xs2 = xs?filter(f)>

If xs is List-like, then this will work. Since the xs List fits into
the memory (although a List can be backed by disk, that's rather
rare), hopefully it's not the kind of data amount that can't fit into
the memory again (as xs2). On the other hand, if xs is an
Iterator-like object, then the above statement fails, with the hint
that xs?filter(f)?sequence would work, but might consumes a lot of
memory.

This is also consistent with how xs[i] works in the existing
FreeMarker versions. That only works if xs is List-like (an FTL
sequence). While xs[i] would be trivial to implement even if xs is
Iterator-like, we don't do that as it's not efficient for a high i,
and so the template author is probably not meant to do that. If he
knows what's he doing though, he can write xs?sequence[i]. Yes, that's
very inefficient if you only use [] once on that sequence, but you see
the logic. map/filter breaks it, as xs?filter(f)[i] works even if xs
is an Iterator, because filter/map currently always returns a
sequence. If xs is Iteartor-like, then I want filter/map to return an
Iterator-like as well, so then [] will fail on it.

As a side note, I will make ?sequence smarter too, so that
xs?sequence[i] won't actually build a sequence if xs is Iterator-like.
It just have to skip the first i elements after all. (The ?sequence is
still required there. It basically says: "I know what I'm doing, treat
this as a sequence.")

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

Am Mo., 1. Apr. 2019 um 00:38 Uhr schrieb Daniel Dekany <ddekany@apache.org
>:

> Monday, February 25, 2019, 10:19:25 AM, Christoph Rüger wrote:
>
> [snip]
> >> > Overall those lambdas allow some great new use cases with pretty
> concise
> >> > syntax. That is good. The ?filter and ?map built-ins are very cool
> >> already.
> >> > If some existing built-ins will be made "smarter" (as you said), then
> it
> >> > will be a great enhancement.
> >> >
> >> > Speaking for us, performance and memory usage is always a concern. So
> it
> >> > would be good to keep an eye on avoiding large new in-memory
> structures.
> >>
> >> Template authors can always do something like
> >> <#assign cheapProducts = prods?filter(...)>, and that will collect
> >> everything into a List internally, as it's eager processing. If an
> >> aggregating ?groupBy is concerning, then this is even more so.
> >
> >
> > Hmm... well.... difficult....
> >
> > So, *<#assign cheapProducts = prods?filter(...)>* will create a new list,
> > while the following does not:
> >
> >  <#list products?filter(it -> it.price < 1000) as product>
> >     ${product.name}
> >   </#list>
> >
> > Right?
> >
> > Earlier you wrote:
> >
> > *"Of course these built-ins aren't specific to #list, they can be
> > usedanywhere. Naturally, they can be chained as well:"*
> >
> > What would be a downside of allowing ?filter / ?map only in #list?
>
> It makes them significantly less useful. A major annoyance that users
> run into in FTL is that you can build a new sequence (wrapped List)
> based on an existing one. They may want to do that, and then assign it
> to a variable, or pass it to a custom macro as argument. (OK, you can,
> with sequence concatenation, but that's very inefficient if you
> exploit to build a sequence element by element.) ?filter and ?map
> would help in lot of those use cases where the users wanted to build a
> new sequence.
>
> Speaking of which, nothing stops users from building gigantic
> sequences from some iterable you provide with sequence concatenation.
> Sure, it's less tempting to do than applying ?filter and ?map, but
> still... users do it.


> > I guess it makes it more complicated to explain.
>
> That too, and also it must be quite off putting to discover a such
> limitation. After all, it's natural to expect that if you can have
> <#list someExp>, then you can factor out someExp into an earlier
> assignment.
>
> Besides, if we really wanted to limit these to #list, then they would
> be the part of the #list syntax, like <#list xs as x where x.foo>.
> Kind of like SQL (yuck).


> > Difficult tradeoffs here :) Sorry sometimes I get confused by jumping
> > between lazy and eager processing.
>
> Well, if you fear users jumping on ?filter/?map outside #list for no
> good enough reason, there can be some option to handle that. But I
> don't think restricting the usage to #list is a good compromise as the
> default.
>

I agree. Just keep as it is.


>
> >> I'm not sure how efficiently could a configuration setting catch these
> >> cases, or if it should be addressed on that level.
> >>
> >
> > Maybe let's postpone configurability discussion a bit until the above is
> > more clear.
>
> In the light of the above, I think we can start thinking about that
> now.
>

On that note on configurability: Would it be possible to programmatically
influence the Collection (Sequence) which is created under the hood?
E.g. by specifying a Factory? I ask because we are using something like
this (
https://dzone.com/articles/a-filebasedcollection-in-java-for-big-collections)
in other places for large collections. I know it is very specific, but just
wanted to bring it up.


> Different... I have added some optimization to ?size. So for example
> if you have <#if list?filter(foo)?size != 0>, then not only it will
> not build any in-memory list (?size never does that, as it only has to
> count), but it only fetches elements until it finds one that matches
> the filter predicate, as at that point the comparison result already
> can be known. This works with comparison with any integer literal, and
> other comparison operators as well.
>
> (Also, list?map(foo)?size basically falls back doing list?size.
> Admittedly that has no much practical application, but still...)
>
>
These are great optimizations.


> --
> Thanks,
>  Daniel Dekany
>
>

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 
5596493Internet: https://synesty.com <https://synesty.com>
Informationen 
zum Datenschutz: https://synesty.com/datenschutz 
<https://synesty.com/datenschutz>

Geschäftsführer: Christoph Rüger

Unternehmenssitz: Jena
Handelsregister B beim Amtsgericht: Jena

Handelsregister-Nummer: HRB 508766
Ust-IdNr.: DE287564982

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

Monday, February 25, 2019, 10:19:25 AM, Christoph Rüger wrote:

[snip]
>> > Overall those lambdas allow some great new use cases with pretty concise
>> > syntax. That is good. The ?filter and ?map built-ins are very cool
>> already.
>> > If some existing built-ins will be made "smarter" (as you said), then it
>> > will be a great enhancement.
>> >
>> > Speaking for us, performance and memory usage is always a concern. So it
>> > would be good to keep an eye on avoiding large new in-memory structures.
>>
>> Template authors can always do something like
>> <#assign cheapProducts = prods?filter(...)>, and that will collect
>> everything into a List internally, as it's eager processing. If an
>> aggregating ?groupBy is concerning, then this is even more so.
>
>
> Hmm... well.... difficult....
>
> So, *<#assign cheapProducts = prods?filter(...)>* will create a new list,
> while the following does not:
>
>  <#list products?filter(it -> it.price < 1000) as product>
>     ${product.name}
>   </#list>
>
> Right?
>
> Earlier you wrote:
>
> *"Of course these built-ins aren't specific to #list, they can be
> usedanywhere. Naturally, they can be chained as well:"*
>
> What would be a downside of allowing ?filter / ?map only in #list?

It makes them significantly less useful. A major annoyance that users
run into in FTL is that you can build a new sequence (wrapped List)
based on an existing one. They may want to do that, and then assign it
to a variable, or pass it to a custom macro as argument. (OK, you can,
with sequence concatenation, but that's very inefficient if you
exploit to build a sequence element by element.) ?filter and ?map
would help in lot of those use cases where the users wanted to build a
new sequence.

Speaking of which, nothing stops users from building gigantic
sequences from some iterable you provide with sequence concatenation.
Sure, it's less tempting to do than applying ?filter and ?map, but
still... users do it.

> I guess it makes it more complicated to explain.

That too, and also it must be quite off putting to discover a such
limitation. After all, it's natural to expect that if you can have
<#list someExp>, then you can factor out someExp into an earlier
assignment.

Besides, if we really wanted to limit these to #list, then they would
be the part of the #list syntax, like <#list xs as x where x.foo>.
Kind of like SQL (yuck).

> Difficult tradeoffs here :) Sorry sometimes I get confused by jumping
> between lazy and eager processing.

Well, if you fear users jumping on ?filter/?map outside #list for no
good enough reason, there can be some option to handle that. But I
don't think restricting the usage to #list is a good compromise as the
default.

>> I'm not sure how efficiently could a configuration setting catch these
>> cases, or if it should be addressed on that level.
>>
>
> Maybe let's postpone configurability discussion a bit until the above is
> more clear.

In the light of the above, I think we can start thinking about that
now.

Different... I have added some optimization to ?size. So for example
if you have <#if list?filter(foo)?size != 0>, then not only it will
not build any in-memory list (?size never does that, as it only has to
count), but it only fetches elements until it finds one that matches
the filter predicate, as at that point the comparison result already
can be known. This works with comparison with any integer literal, and
other comparison operators as well.

(Also, list?map(foo)?size basically falls back doing list?size.
Admittedly that has no much practical application, but still...)

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

Am Mo., 25. Feb. 2019 um 00:46 Uhr schrieb Daniel Dekany <ddekany@apache.org
>:

> Sunday, February 24, 2019, 11:05:52 PM, Christoph Rüger wrote:
>
> > Am So., 24. Feb. 2019 um 19:40 Uhr schrieb Daniel Dekany <
> ddekany@apache.org
> >>:
> >
> >> Sunday, February 24, 2019, 12:45:48 AM, Christoph Rüger wrote:
> >>
> >> > Thanks you very much. This looks great.
> >> > Our testsuite passes all freemarker related tests.
> >> >
> >> > Also some simple manual tests were successful e.g.
> >> > <#list row.cols?map(col -> col.title)?filter(ct ->
> ct?contains("name"))
> >> as
> >> coltitle >>${coltitle!}<#sep>,</#sep></#list>
> >> > <#list description?split(" ")?filter(word -> word == "test") as
> >> filteredWord>>${filteredWord}</#list>
> >> >
> >> > Regarding other use-cases for lazy-evaluation:
> >> >
> >> > Do you think a ?count (as terminal operation) without building up a
> list
> >> in
> >> > memory?
> >> > E.g.
> >> > Number of products < 1000: ${products?filter(it -> it.price <
> >> 1000)?count}
> >> >
> >> > (I tested ${products?filter(it -> it.price < 1000)?size} but I think
> this
> >> > builds up a list in memory which can be large)
> >>
> >> Good point! We don't need to introduce ?count for this though, simply
> >> ?size has to be made smarter.
> >>
> >
> > Right, that's better. Let's use ?size.
> >
> >> > Or what about other aggregations for numerical values like sum, avg,
> min,
> >> > max e.g.:
> >> > Cheapest price of products < 1000: ${products?filter(it -> it.price <
> >> > 1000)?map(it -> it.price)?min}
> >>
> >> We have ?min and ?max, they also need to be made smarter.
> >
> >
> > Ok. If it can be done efficiently then good. As ?min ?max are already
> there
> > I guess people would expect to use them...as I did.
>
> After writing the previous mail I have realized that ?min, ?max,
> and also ?first, ?seq_contains, and ?seq_index_of already enable lazy
> processing.
>
> >> As of ?sum
> >> and ?acg... they are doable. But aggregates are mostly useful combined
> >> with "group by". Technically, we can do all that, but I'm not 100%
> >> sure if it's wise to do.
> >
> > The more such operations the template
> >> language supports, the more people will tend to move such calculations
> >> from the Java code that builds the data-model to the template. But the
> >> good practice was always the opposite of that: only do presentation
> >> logic in templates, not "business" calculations. But then again, there
> >> are the situations when the template authors have little control over
> >> the data model (typically, when you make a report/mail template for
> >> some business application that was developed by another party), and
> >> then they are happy that they can get the work done in the template.
> >> So, it's not easy to decide which compromise is the better.
> >>
> >
> > Yeah difficult decision.... We are such an application you mention where
> > the template is the only tool available.
> >
> > Maybe groupBy and aggregates like ?sum, ?avg should not be part of it for
> > now. I think Grouping requires keeping data in memory... that's pretty
> > powerful but performance critical. Or have a way to enable/disable it
> with
> > configuration.
> >
> > Speaking of configuration / customization: We have built something like
> > groupBy ourselves outside the template language (for a special table
> > object). We added some "safety" stuff to abort processing if the data
> grows
> > too large over a defined threshold. It would be good to be able to
> > customize this too.
>
> Often such thing has the best place in the data-model, as you know
> more about your application than the template engine. (Not to mention
> when you could do the group-by in SQL, but if the template can do it
> as well, maybe the template authors just use that, and so the backend
> guys will never know about this requirement.) Also there's the
> inherent inefficiency with built-ins that they have to work with
> TemplateModel-s, while a solution in the model probably can work
> directly with the native objects. So I'm not sure if we want
> aggregating groupBy in the template language.
>

Right.
It's probably better to not have ?groupBy in the template language and do
it in the data-model.


>
> >> BTW, the kind of "group by" operation that doesn't aggregate anything,
> >> just splits the collection into smaller collections for each distinct
> >> "group by" value, like Collectors.groupBy in Java 8, would be often
> >> useful for presentation logic. The use case is that you get a List of
> >> rows, and let's say you have a Year column in there, and the List is
> >> ordered by that. Then you should render it so that you don't show the
> >> same year again and again. Like if it's a HTML table then you are
> >> supposed to use td.@rowspan in the year column. It's quite awkward to
> >> do such thing currently (try it and you will see...), despite that
> >> it's a common requirement and is clearly a presentation decision, not
> >> really "business logic". So, we could introduce a built-in for that:
> >> collection?splitBy(lambdaOrFunctionOrMethod). I would avoid calling it
> >> groupBy, since that has a different meaning in SQL (the aggregating
> >> groupBy veriant). So now rowspan by year would be like:
> >>
> >>   <table>
> >>     <tr>
> >>       <th>Year</th>
> >>       <th>Department</th>
> >>       <th>Income</th>
> >>     </tr>
> >>     <#list rows?groupBy(row -> row.year) as sameYearRows>
>
> I meant ?splitBy...
>

Sorry, forgot about it while writing.


>
> >>       <#list sameYearRows as row>
> >>         <tr>
> >>           <#if row?isFirst>
> >>             <td rowspan="${sameYearRows?size}">${row.year}</td>
> >>           </#if>
> >>           <td>${row.department}</td>
> >>           <td>${row.income}</td>
> >>         </tr>
> >>       </#list>
> >>     <#list>
> >>   </table>
> >>
> >> Or if you have to print a new H2 and table for each year:
> >>
> >>   <#list rows?groupBy(row -> row.year) as sameYearRows>
>
> Again, ?splitBy


> >>     <h2>${sameYearRows[0].year}<h2>
> >>     <table>
> >>       <tr>
> >>         <th>Department</th>
> >>         <th>Income</th>
> >>       </tr>
> >>       <#list sameYearRows as row>
> >>         <tr>
> >>           <td>${row.department}</td>
> >>           <td>${row.income}</td>
> >>         </tr>
> >>       </#list>
> >>     </table>
> >>   <#list>
> >>
> >>
> > Yes that can be handy. But the implementation should be careful to avoid
> > large in-memory structures for the "groups"
> > Does ?groupBy(row -> row.year) create a new List for each group? Or is
> > it kind of a "Views" on the original list?
>
> It has to collect the whole thing into the memory, as we can't assume
> that the source collection is sorted by the grouping key. So maybe it
> should instead start a new group when the grouping key value changes
> in the stream of elements. Then the name should be "splitAtChanges":
> <#list rows?splitAtChanges(row -> row.year) as sameYearRows>


> >> Lambdas happen to be handy with this as well. Actually, some long
> >> existing built-ins, like ?sortBy, ?contains, etc. could use them as
> >> well (a bit tricky for them technically due to backward compatibility
> >> requirements, but I belive I can work that around.)
> >>
> >
> > Overall those lambdas allow some great new use cases with pretty concise
> > syntax. That is good. The ?filter and ?map built-ins are very cool
> already.
> > If some existing built-ins will be made "smarter" (as you said), then it
> > will be a great enhancement.
> >
> > Speaking for us, performance and memory usage is always a concern. So it
> > would be good to keep an eye on avoiding large new in-memory structures.
>
> Template authors can always do something like
> <#assign cheapProducts = prods?filter(...)>, and that will collect
> everything into a List internally, as it's eager processing. If an
> aggregating ?groupBy is concerning, then this is even more so.


Hmm... well.... difficult....

So, *<#assign cheapProducts = prods?filter(...)>* will create a new list,
while the following does not:

 <#list products?filter(it -> it.price < 1000) as product>
    ${product.name}
  </#list>

Right?

Earlier you wrote:

*"Of course these built-ins aren't specific to #list, they can be
usedanywhere. Naturally, they can be chained as well:"*

What would be a downside of allowing ?filter / ?map only in #list?

I guess it makes it more complicated to explain.
Difficult tradeoffs here :) Sorry sometimes I get confused by jumping
between lazy and eager processing.



> I'm not sure how efficiently could a configuration setting catch these
> cases, or if it should be addressed on that level.
>

Maybe let's postpone configurability discussion a bit until the above is
more clear.



>
> > My thoughts so far.
> > Thanks
> > Christoph
> >
> >
> >>
> >> > Didn't think this through, but just had the idea while testing.
> >> >
> >> > Anyway, good work!
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Am Sa., 23. Feb. 2019 um 18:19 Uhr schrieb Daniel Dekany <
> >> ddekany@apache.org
> >> >>:
> >> >
> >> >> I have pushed an implementation of the restricted lambdas we were
> >> >> talking about. Guys, please review/test it.
> >> >>
> >> >> I call these "local lambdas" in the source code (but I'm open for
> >> >> suggestions), as the function they define can only be called in the
> >> >> same variable scope where the lambda was (as we have no closures in
> >> >> the template language, nor final variables). Also, they can only be
> >> >> used as the parameters of the built-ins that explicitly support them.
> >> >> As the subject shows, the main goal was just support filtering for
> >> >> #list, without a nested #if, as that breaks #sep, #else, it?hasNext,
> >> >> it?index, etc. (I will try to make more universal lambdas in FM3... I
> >> >> guess it would be way too tricky in FM2.)
> >> >>
> >> >> For now, I have only added two built-ins that support lambdas:
> ?filter
> >> >> and ?map. Any ideas what other such built-ins would be often useful
> in
> >> >> templates (with use case, if possible)?
> >> >>
> >> >> Examples of using ?filter and ?map:
> >> >>
> >> >>   <#list products?filter(it -> it.price < 1000) as product>
> >> >>     ${product.name}
> >> >>   </#list>
> >> >>
> >> >>   <#list products?map(it -> it.name) as name>
> >> >>     ${name}
> >> >>   </#list>
> >> >>
> >> >> Of course these built-ins aren't specific to #list, they can be used
> >> >> anywhere. Naturally, they can be chained as well:
> >> >>
> >> >>   <#assign chepProdNames = products
> >> >>      ?filter(it -> it.price < 1000)
> >> >>      ?map(it -> it.name)
> >> >>   >
> >> >>
> >> >> As a side note, ?filter and ?map also accepts FTL function-s and Java
> >> >> methods as its parameter, not only lambdas.
> >> >>
> >> >> A tricky aspect of this feature is lazy evaluation, in similar sense
> >> >> as Java 8 Stream intermediate operations are lazy. On most places,
> >> >> ?filer and ?map are eager, that is, they return a completed sequence
> >> >> of items. That's because our restricted lambdas only work correctly
> >> >> "locally". However, there are very common situations where it's clear
> >> >> that we can use lazy ("streaming") evaluation, as we know where the
> >> >> resulting stream of elements will be consumed:
> >> >>
> >> >> - One such case is when these kind of built-ins are chained. Like in
> >> >>   the last example, ?filter doesn't construct a List in memory,
> >> >>   instead the elements just flow through it (some is dropped, as it's
> >> >>   filtering), into ?map. Only the ?map at the end of the chain will
> >> >>   built a List eagerly. Some other built-ins also allow the left-hand
> >> >>   built-in to stream, like in ?filter(...)?map(...)?join(", ") no
> List
> >> >>   is built anywhere, instead the elements flow through both ?filter
> >> >>   and ?map, and ?join just appends them to the StringBuilder where it
> >> >>   creates its results.
> >> >>
> >> >> - #list also enables lazy evaluation to its 1st parameter. So in the
> >> >>   #list examples above, yet again no List is built in memory.
> >> >>
> >> >> - There are other such cases, which I didn't implement yet. For
> >> >>   example the sequence slice operator, like, xs?filter(f)[10..20],
> >> >>   should allow lazy evaluation.
> >> >>
> >> >> Feedback is welcome!
> >> >>
> >> >>
> >> >> Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
> >> >>
> >> >> > Hey Daniel,
> >> >> > I'm very sorry, but I didn't make any progress with this. I think I
> >> was a
> >> >> > bit over-motivated, but unfortunately I cannot spend more time on
> this
> >> >> for
> >> >> > various reasons.
> >> >> > You can take this over. I'm glad to help out with testing and
> >> feedback.
> >> >> >
> >> >> > Thanks
> >> >> > Christoph
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <
> >> >> ddekany@apache.org
> >> >> >>:
> >> >> >
> >> >> >> Any progress in this? I think I will give it a try in the coming
> days
> >> >> >> otherwise.
> >> >> >>
> >> >> >>
> >> >> >> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
> >> >> >>
> >> >> >> > See my answers inline...
> >> >> >> >
> >> >> >> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
> >> >> >> >
> >> >> >> >> Thanks Daniel for your feedback. See my answers below
> >> >> >> >>
> >> >> >> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
> >> >> >> ddekany@apache.org
> >> >> >> >>>:
> >> >> >> >>
> >> >> >> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
> >> >> >> >>>
> >> >> >> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
> >> >> >> >>> ddekany@apache.org
> >> >> >> >>> >>:
> >> >> >> >>> >
> >> >> >> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet
> wrote:
> >> >> >> >>> >>
> >> >> >> >>> >> > Hi,
> >> >> >> >>> >> >
> >> >> >> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <
> >> >> c.rueger@synesty.com>
> >> >> >> a
> >> >> >> >>> >> écrit :
> >> >> >> >>> >> >
> >> >> >> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
> >> >> >> >>> >> ddekany@apache.org
> >> >> >> >>> >> > :
> >> >> >> >>> >> >
> >> >> >> >>> >> > It's certainly tricky, but as far as I see possible (but
> >> then,
> >> >> who
> >> >> >> >>> >> >
> >> >> >> >>> >> > knows what will one find when actually working on it).
> It's
> >> >> also a
> >> >> >> >>> >> > feature missing a lot. It's especially missing for #list
> (I
> >> >> know
> >> >> >> that
> >> >> >> >>> >> > you need it for something else), because if you filter
> the
> >> >> items
> >> >> >> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will
> not
> >> be
> >> >> >> usable.
> >> >> >> >>> >> >
> >> >> >> >>> >> > Let me say that I disagree here.
> >> >> >> >>> >> >
> >> >> >> >>> >> > I do not think that closures are required for FreeMarker,
> >> nor
> >> >> that
> >> >> >> >>> they
> >> >> >> >>> >> are a good idea.
> >> >> >> >>> >> >
> >> >> >> >>> >> > If we add new features to the FreeMarker *tempate
> engine* I
> >> >> would
> >> >> >> >>> >> > rather we focus on multi-part macro body rather than an
> >> >> advanced
> >> >> >> >>> >> language feature like closures.
> >> >> >> >>> >> >
> >> >> >> >>> >> > You can add ?filter and ?map if you want, a simple
> >> expression
> >> >> as
> >> >> >> >>> >> parameter should be enough.
> >> >> >> >>> >>
> >> >> >> >>> >> Yes, as I said, we certainly start with only allowing
> lambdas
> >> in
> >> >> >> >>> >> ?filter/?map, also certainly in ?contains.
> >> >> >> >>> >>
> >> >> >> >>> > Would be enough in my opinion and very useful.
> >> >> >> >>> >
> >> >> >> >>> > Is it possiblefor you to give some pointers to the code on
> how
> >> >> this
> >> >> >> could
> >> >> >> >>> > be implemented? I would maybe like to wrap my head around
> this
> >> a
> >> >> >> little
> >> >> >> >>> bit.
> >> >> >> >>>
> >> >> >> >>> Please feel yourself encouraged! (:
> >> >> >> >>>
> >> >> >> >>> > I started looking at seq_containsBI (
> >> >> >> >>> >
> >> >> >> >>>
> >> >> >>
> >> >>
> >>
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
> >> >> >> >>> )
> >> >> >> >>> > and
> >> >> >> >>> > and reverseBI (
> >> >> >> >>> >
> >> >> >> >>>
> >> >> >>
> >> >>
> >>
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
> >> >> >> >>> )
> >> >> >> >>> > just to find something related (seq_containsBI checks
> >> something)
> >> >> and
> >> >> >> >>> > reverseBI returns a new sequence.
> >> >> >> >>> > What I haven't found is a function which takes an Expression
> >> as a
> >> >> >> >>> > parameter.
> >> >> >> >>> > Is there something similar already or would that be a new
> >> thing?
> >> >> >> >>>
> >> >> >> >>> It's a new thing in that it will be part of the expression
> syntax
> >> >> >> >>> (even if for now we will only allow lambdas as the parameters
> of
> >> a
> >> >> few
> >> >> >> >>> built-ins, so that we can get away without closures). So it's
> a
> >> new
> >> >> >> >>> Expression subclass, and has to be part of the parser
> (ftl.jj) as
> >> >> >> >>> well.
> >> >> >> >>
> >> >> >> >> Hmm, that parser stuff is new for me, it'll take me some time
> to
> >> get
> >> >> >> into
> >> >> >> >> it.
> >> >> >> >
> >> >> >> > So it's a JavaCC lexer+parser. With a few twists... sorry, but
> this
> >> >> >> > code has history... :)
> >> >> >> >
> >> >> >> >>> As of lazy evaluation of parameters expressions, that's
> already
> >> >> >> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and
> you
> >> >> will
> >> >> >> >>> see it's trivial to do, but the situation there is much
> simpler.
> >> >> >> >>>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>>
> >> >> >> >>> In principle, a LambdaExpression should evaluate to a
> >> >> >> >>> TemplateMethodModelEx, and then you pass that
> >> TemplateMethodModelEx
> >> >> to
> >> >> >> >>> the called built-in or whatever it is. But with the approach
> of
> >> >> >> >>> BuiltInsWithParseTimeParameters we can certainly even skip
> that,
> >> and
> >> >> >> >>> just bind to the LambdaExpression directly, add a LocalContext
> >> that
> >> >> >> >>> contains the lambda arguments, end evaluate the
> LambdaExpression
> >> >> right
> >> >> >> >>> there, in the built-in implementation. Or at least at a very
> >> quick
> >> >> >> >>> glance I think so.
> >> >> >> >>>
> >> >> >> >> Not sure I can follow completely but that hint with*
> >> >> >> >> BuiltInsWithParseTimeParameters* got me started, but at the
> moment
> >> >> I'm
> >> >> >> >> stuck as I need to get more familiar with the internals of
> >> >> Freemarker.
> >> >> >> I am
> >> >> >> >> also not sure I am on the same page regarding the syntax we are
> >> >> aiming
> >> >> >> for
> >> >> >> >> and why I would need to extend the parser when there is
> something
> >> >> like
> >> >> >> >> BuiltInsWithParseTimeParameters....
> >> >> >> >
> >> >> >> > I assumed that the syntax will be similar to the Java lambda
> syntax
> >> >> >> > (see below why), and that's of course needs lexer/parser
> changes.
> >> >> >> >
> >> >> >> >> Here is an example what I have in mind:
> >> >> >> >> I started with the ?filter() builtin. I had a syntax like this
> in
> >> >> mind:
> >> >> >> >>
> >> >> >> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
> >> >> >> >> *Example 2: ["a","b","c"]?filter(element, element ==
> >> >> someOtherVariable,
> >> >> >> >> someOtherVariable="c")*
> >> >> >> >
> >> >> >> > Looking at the above one believe that the value of `element` is
> >> passed
> >> >> >> > in as first argument, and the the value of `element == "c"` as
> the
> >> >> >> > second, but that's not the case. It's much better if it's
> visible
> >> that
> >> >> >> > you got some kind of anonymous function definition there without
> >> >> >> > knowing about the "filter" built-in.
> >> >> >> >
> >> >> >> > Java already has a syntax for expressing this kind of thing, so
> it
> >> >> >> > would be better to use that familiar syntax. As far as I know it
> >> >> >> > doesn't conflict with ours (it kind of it does, but not
> fatally).
> >> >> >> >
> >> >> >> > Also, even if for now we only allow this in said built-ins, we
> >> >> >> > shouldn't exclude the possibility of making this kind of
> expression
> >> >> >> > accessible elsewhere as well. Then, on many places the parsed
> won't
> >> >> >> > know what kind of value is expected (consider passing a lambda
> to
> >> an
> >> >> >> > user defined directive for example), so a syntax like above
> won't
> >> >> >> > work.
> >> >> >> >
> >> >> >> >> Not sure if that's what you have in mind too, but to me it made
> >> sense
> >> >> >> with
> >> >> >> >> regards to BuiltInsWithParseTimeParameters and I could start
> >> without
> >> >> >> >> touching parser stuff.
> >> >> >> >
> >> >> >> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only
> a
> >> >> >> > bit...), as a call to a such built-in looks like a call to any
> >> other
> >> >> >> > built-in.
> >> >> >> >
> >> >> >> >> *1st argument 'element'* would just be the iterator variable
> >> similar
> >> >> to
> >> >> >> >> <#list ["a","b","c"] as *element*>
> >> >> >> >> 2nd argument is the filter lambda expression... aka our filter
> >> >> condition
> >> >> >> >> 3rd+n argument are optional parameters in case used in the
> lambda
> >> >> >> expression
> >> >> >> >
> >> >> >> > #list is special, similarly as `for ... in ...` is in most
> >> languages.
> >> >> >> > It's not lambda-ish either; you can't write `as element/2` or
> such.
> >> >> >> >
> >> >> >> >> So at first I was looking how <#list> works and found
> >> IteratorBlock,
> >> >> and
> >> >> >> >> though I could reuse it somehow.
> >> >> >> >>
> >> >> >> >> Here is some simple pseudo code I played around for the for
> >> Example
> >> >> 1:
> >> >> >> >>
> >> >> >> >> static class filter_BI extends BuiltInWithParseTimeParameters {
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>         TemplateModel _eval(Environment env) throws
> >> >> TemplateException {
> >> >> >> >>
> >> >> >> >>             // sequence
> >> >> >> >>
> >> >> >> >>         TemplateModel targetValue =
> target.evalToNonMissing(env);
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>             List parameters = this.parameters;
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>             Expression iteratorAlias = (Expression)
> >> >> parameters.get(0);
> >> >> >> >>
> >> >> >> >>             Expression conditionExpression = (Expression)
> >> >> >> parameters.get(1);
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>             TemplateSequenceModel seq =
> (TemplateSequenceModel)
> >> >> >> target.eval
> >> >> >> >> (env);
> >> >> >> >>
> >> >> >> >>             for (int i = 0; i < seq.size(); i++) {
> >> >> >> >>
> >> >> >> >>                TemplateModel cur = seq.get(i);
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>                // this is where I am stuck at the moment
> >> >> >> >>
> >> >> >> >>                // I basically want to evaluate
> conditionExpression
> >> >> >> >> where iteratorAlias
> >> >> >> >> is basically what I passed as 'element'
> >> >> >> >>
> >> >> >> >>                // I am not sure if or how LocalContext could
> come
> >> >> into
> >> >> >> play
> >> >> >> >> here
> >> >> >> >>
> >> >> >> >>                // basically for each iteration I would assign
> the
> >> >> >> current
> >> >> >> >> loop element to a context variable with the name 'element'
> >> >> >> >>
> >> >> >> >>                // and then evaluate conditionExpression with
> that
> >> >> >> context.
> >> >> >> >>
> >> >> >> >>                // if conditionExpression is "true" then I would
> >> >> populate
> >> >> >> >> add the current sequence element 'cur'
> >> >> >> >>
> >> >> >> >>                // to a new result-List.... and return that....
> >> >> something
> >> >> >> >>
> >> >> >> >>                // I wanted to reuse IteratorBlock here somehow,
> >> but
> >> >> >> didn't
> >> >> >> >> get it to work yet.
> >> >> >> >>
> >> >> >> >>                // maybe this is a stupid idea, or we just need
> >> >> something
> >> >> >> >> similar
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>             }
> >> >> >> >>
> >> >> >> >> }
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Ok so far for my pseudo code....  Maybe you could give some
> more
> >> >> >> pointers
> >> >> >> >> based on that... in case this makes any sense ...
> >> >> >> >
> >> >> >> > For LocalContext, Environment has a pushLocalContext method. See
> >> the
> >> >> >> > calls to it, like in visit(TemplateElement[],
> >> TemplateDirectiveModel,
> >> >> >> > Map, List).
> >> >> >> >
> >> >> >> > To avoid running into more new things at once than necessary,
> >> perhaps
> >> >> >> > you should start with seq?seq_contains(lambda). The end result
> >> should
> >> >> >> > be like:
> >> >> >> >
> >> >> >> >   users?contains(u -> u.paying)
> >> >> >> >
> >> >> >> >>> Another similarity to BuiltInsWithParseTimeParameters is that
> we
> >> >> won't
> >> >> >> >>> allow separating the `?someBI` and `(arg)`. Like, with the
> >> example
> >> >> of
> >> >> >> >>> `cond?then(1, 2)`, you aren't allowed to do this:
> >> >> >> >>>
> >> >> >> >>>   <#assign t=cond?then>
> >> >> >> >>>   ${t(1, 2)}
> >> >> >> >>>
> >> >> >> >>> That maybe looks natural, but most other built-ins allow
> that. Of
> >> >> >> >>> course we can't allow that for ?filter etc., because then we
> need
> >> >> >> >>> closures again.
> >> >> >> >>>
> >> >> >> >>> >> Multi-part macro body is also planned. Means, I know it
> >> >> definitely
> >> >> >> >>> >> should be added, but who knows when that's done... I mean,
> >> it's
> >> >> like
> >> >> >> >>> >> that for what, a decade? (: It's not even decided what it
> >> exactly
> >> >> >> >>> >> does, as there are many ways of approaching this. (I have
> my
> >> own
> >> >> >> idea
> >> >> >> >>> >> about what the right compromise would be, but others has
> other
> >> >> >> >>> >> ideas...)
> >> >> >> >>> >>
> >> >> >> >>> >> Filtering lists bothers me because the template language
> >> should
> >> >> be
> >> >> >> >>> >> (and somewhat indeed is) specialized on listing things on
> >> fancy
> >> >> ways
> >> >> >> >>> >> that used to come up when generating document-like output.
> >> (If it
> >> >> >> >>> >> doesn't do things like that, you might as well use a
> general
> >> >> purpose
> >> >> >> >>> >> language.) Thus, that filter is unsolved (filtering with
> #if
> >> is
> >> >> >> >>> >> verbose and spoils #sep etc.) bothers me a lot.
> >> >> >> >>> >>
> >> >> >> >>> >> BTW, ?filter and ?map is also especially handy in our case
> as
> >> >> >> >>> >> FreeMarker doesn't support building new sequences
> (sequences
> >> are
> >> >> >> >>> >> immutable). Although it has sequence concatenation with
> `+`,
> >> it's
> >> >> >> not
> >> >> >> >>> >> good for building a sequence one by one, unless the
> sequence
> >> >> will be
> >> >> >> >>> >> quite short.
> >> >> >> >>> >>
> >> >> >> >>> > Good point.
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> >>
> >> >> >> >>> >> > Cheers,
> >> >> >> >>> >> > -- Denis.
> >> >> >> >>> >>
> >> >> >> >>> >> --
> >> >> >> >>> >> Thanks,
> >> >> >> >>> >>  Daniel Dekany
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> Thanks,
> >> >> >> >>>  Daniel Dekany
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >> Thanks
> >> >> >> >> Christoph
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >> --
> >> >> >> Thanks,
> >> >> >>  Daniel Dekany
> >> >> >>
> >> >> >>
> >> >> >
> >> >> > --
> >> >> > Christoph Rüger, Geschäftsführer
> >> >> > Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
> >> >> > Programmieren - Automatisierung, Schnittstellen, Datenfeeds
> >> >> >
> >> >> > Xing: https://www.xing.com/profile/Christoph_Rueger2
> >> >> > LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
> >> >> >
> >> >>
> >> >> --
> >> >> Thanks,
> >> >>  Daniel Dekany
> >> >>
> >> >>
> >> >
> >>
> >> --
> >> Thanks,
> >>  Daniel Dekany
> >>
> >>
> >
>
> --
> Thanks,
>  Daniel Dekany
>
>

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 
5596493Internet: https://synesty.com <https://synesty.com>
Informationen 
zum Datenschutz: https://synesty.com/datenschutz 
<https://synesty.com/datenschutz>

Geschäftsführer: Christoph Rüger

Unternehmenssitz: Jena
Handelsregister B beim Amtsgericht: Jena

Handelsregister-Nummer: HRB 508766
Ust-IdNr.: DE287564982

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

Sunday, February 24, 2019, 11:05:52 PM, Christoph Rüger wrote:

> Am So., 24. Feb. 2019 um 19:40 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>:
>
>> Sunday, February 24, 2019, 12:45:48 AM, Christoph Rüger wrote:
>>
>> > Thanks you very much. This looks great.
>> > Our testsuite passes all freemarker related tests.
>> >
>> > Also some simple manual tests were successful e.g.
>> > <#list row.cols?map(col -> col.title)?filter(ct -> ct?contains("name"))
>> as
>> coltitle >>${coltitle!}<#sep>,</#sep></#list>
>> > <#list description?split(" ")?filter(word -> word == "test") as
>> filteredWord>>${filteredWord}</#list>
>> >
>> > Regarding other use-cases for lazy-evaluation:
>> >
>> > Do you think a ?count (as terminal operation) without building up a list
>> in
>> > memory?
>> > E.g.
>> > Number of products < 1000: ${products?filter(it -> it.price <
>> 1000)?count}
>> >
>> > (I tested ${products?filter(it -> it.price < 1000)?size} but I think this
>> > builds up a list in memory which can be large)
>>
>> Good point! We don't need to introduce ?count for this though, simply
>> ?size has to be made smarter.
>>
>
> Right, that's better. Let's use ?size.
>
>> > Or what about other aggregations for numerical values like sum, avg, min,
>> > max e.g.:
>> > Cheapest price of products < 1000: ${products?filter(it -> it.price <
>> > 1000)?map(it -> it.price)?min}
>>
>> We have ?min and ?max, they also need to be made smarter.
>
>
> Ok. If it can be done efficiently then good. As ?min ?max are already there
> I guess people would expect to use them...as I did.

After writing the previous mail I have realized that ?min, ?max,
and also ?first, ?seq_contains, and ?seq_index_of already enable lazy
processing.

>> As of ?sum
>> and ?acg... they are doable. But aggregates are mostly useful combined
>> with "group by". Technically, we can do all that, but I'm not 100%
>> sure if it's wise to do.
>
> The more such operations the template
>> language supports, the more people will tend to move such calculations
>> from the Java code that builds the data-model to the template. But the
>> good practice was always the opposite of that: only do presentation
>> logic in templates, not "business" calculations. But then again, there
>> are the situations when the template authors have little control over
>> the data model (typically, when you make a report/mail template for
>> some business application that was developed by another party), and
>> then they are happy that they can get the work done in the template.
>> So, it's not easy to decide which compromise is the better.
>>
>
> Yeah difficult decision.... We are such an application you mention where
> the template is the only tool available.
>
> Maybe groupBy and aggregates like ?sum, ?avg should not be part of it for
> now. I think Grouping requires keeping data in memory... that's pretty
> powerful but performance critical. Or have a way to enable/disable it with
> configuration.
>
> Speaking of configuration / customization: We have built something like
> groupBy ourselves outside the template language (for a special table
> object). We added some "safety" stuff to abort processing if the data grows
> too large over a defined threshold. It would be good to be able to
> customize this too.

Often such thing has the best place in the data-model, as you know
more about your application than the template engine. (Not to mention
when you could do the group-by in SQL, but if the template can do it
as well, maybe the template authors just use that, and so the backend
guys will never know about this requirement.) Also there's the
inherent inefficiency with built-ins that they have to work with
TemplateModel-s, while a solution in the model probably can work
directly with the native objects. So I'm not sure if we want
aggregating groupBy in the template language.

>> BTW, the kind of "group by" operation that doesn't aggregate anything,
>> just splits the collection into smaller collections for each distinct
>> "group by" value, like Collectors.groupBy in Java 8, would be often
>> useful for presentation logic. The use case is that you get a List of
>> rows, and let's say you have a Year column in there, and the List is
>> ordered by that. Then you should render it so that you don't show the
>> same year again and again. Like if it's a HTML table then you are
>> supposed to use td.@rowspan in the year column. It's quite awkward to
>> do such thing currently (try it and you will see...), despite that
>> it's a common requirement and is clearly a presentation decision, not
>> really "business logic". So, we could introduce a built-in for that:
>> collection?splitBy(lambdaOrFunctionOrMethod). I would avoid calling it
>> groupBy, since that has a different meaning in SQL (the aggregating
>> groupBy veriant). So now rowspan by year would be like:
>>
>>   <table>
>>     <tr>
>>       <th>Year</th>
>>       <th>Department</th>
>>       <th>Income</th>
>>     </tr>
>>     <#list rows?groupBy(row -> row.year) as sameYearRows>

I meant ?splitBy...

>>       <#list sameYearRows as row>
>>         <tr>
>>           <#if row?isFirst>
>>             <td rowspan="${sameYearRows?size}">${row.year}</td>
>>           </#if>
>>           <td>${row.department}</td>
>>           <td>${row.income}</td>
>>         </tr>
>>       </#list>
>>     <#list>
>>   </table>
>>
>> Or if you have to print a new H2 and table for each year:
>>
>>   <#list rows?groupBy(row -> row.year) as sameYearRows>

Again, ?splitBy

>>     <h2>${sameYearRows[0].year}<h2>
>>     <table>
>>       <tr>
>>         <th>Department</th>
>>         <th>Income</th>
>>       </tr>
>>       <#list sameYearRows as row>
>>         <tr>
>>           <td>${row.department}</td>
>>           <td>${row.income}</td>
>>         </tr>
>>       </#list>
>>     </table>
>>   <#list>
>>
>>
> Yes that can be handy. But the implementation should be careful to avoid
> large in-memory structures for the "groups"
> Does ?groupBy(row -> row.year) create a new List for each group? Or is
> it kind of a "Views" on the original list?

It has to collect the whole thing into the memory, as we can't assume
that the source collection is sorted by the grouping key. So maybe it
should instead start a new group when the grouping key value changes
in the stream of elements. Then the name should be "splitAtChanges":
<#list rows?splitAtChanges(row -> row.year) as sameYearRows>

>> Lambdas happen to be handy with this as well. Actually, some long
>> existing built-ins, like ?sortBy, ?contains, etc. could use them as
>> well (a bit tricky for them technically due to backward compatibility
>> requirements, but I belive I can work that around.)
>>
>
> Overall those lambdas allow some great new use cases with pretty concise
> syntax. That is good. The ?filter and ?map built-ins are very cool already.
> If some existing built-ins will be made "smarter" (as you said), then it
> will be a great enhancement.
>
> Speaking for us, performance and memory usage is always a concern. So it
> would be good to keep an eye on avoiding large new in-memory structures.

Template authors can always do something like
<#assign cheapProducts = prods?filter(...)>, and that will collect
everything into a List internally, as it's eager processing. If an
aggregating ?groupBy is concerning, then this is even more so.

I'm not sure how efficiently could a configuration setting catch these
cases, or if it should be addressed on that level.

> My thoughts so far.
> Thanks
> Christoph
>
>
>>
>> > Didn't think this through, but just had the idea while testing.
>> >
>> > Anyway, good work!
>> >
>> >
>> >
>> >
>> >
>> >
>> > Am Sa., 23. Feb. 2019 um 18:19 Uhr schrieb Daniel Dekany <
>> ddekany@apache.org
>> >>:
>> >
>> >> I have pushed an implementation of the restricted lambdas we were
>> >> talking about. Guys, please review/test it.
>> >>
>> >> I call these "local lambdas" in the source code (but I'm open for
>> >> suggestions), as the function they define can only be called in the
>> >> same variable scope where the lambda was (as we have no closures in
>> >> the template language, nor final variables). Also, they can only be
>> >> used as the parameters of the built-ins that explicitly support them.
>> >> As the subject shows, the main goal was just support filtering for
>> >> #list, without a nested #if, as that breaks #sep, #else, it?hasNext,
>> >> it?index, etc. (I will try to make more universal lambdas in FM3... I
>> >> guess it would be way too tricky in FM2.)
>> >>
>> >> For now, I have only added two built-ins that support lambdas: ?filter
>> >> and ?map. Any ideas what other such built-ins would be often useful in
>> >> templates (with use case, if possible)?
>> >>
>> >> Examples of using ?filter and ?map:
>> >>
>> >>   <#list products?filter(it -> it.price < 1000) as product>
>> >>     ${product.name}
>> >>   </#list>
>> >>
>> >>   <#list products?map(it -> it.name) as name>
>> >>     ${name}
>> >>   </#list>
>> >>
>> >> Of course these built-ins aren't specific to #list, they can be used
>> >> anywhere. Naturally, they can be chained as well:
>> >>
>> >>   <#assign chepProdNames = products
>> >>      ?filter(it -> it.price < 1000)
>> >>      ?map(it -> it.name)
>> >>   >
>> >>
>> >> As a side note, ?filter and ?map also accepts FTL function-s and Java
>> >> methods as its parameter, not only lambdas.
>> >>
>> >> A tricky aspect of this feature is lazy evaluation, in similar sense
>> >> as Java 8 Stream intermediate operations are lazy. On most places,
>> >> ?filer and ?map are eager, that is, they return a completed sequence
>> >> of items. That's because our restricted lambdas only work correctly
>> >> "locally". However, there are very common situations where it's clear
>> >> that we can use lazy ("streaming") evaluation, as we know where the
>> >> resulting stream of elements will be consumed:
>> >>
>> >> - One such case is when these kind of built-ins are chained. Like in
>> >>   the last example, ?filter doesn't construct a List in memory,
>> >>   instead the elements just flow through it (some is dropped, as it's
>> >>   filtering), into ?map. Only the ?map at the end of the chain will
>> >>   built a List eagerly. Some other built-ins also allow the left-hand
>> >>   built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
>> >>   is built anywhere, instead the elements flow through both ?filter
>> >>   and ?map, and ?join just appends them to the StringBuilder where it
>> >>   creates its results.
>> >>
>> >> - #list also enables lazy evaluation to its 1st parameter. So in the
>> >>   #list examples above, yet again no List is built in memory.
>> >>
>> >> - There are other such cases, which I didn't implement yet. For
>> >>   example the sequence slice operator, like, xs?filter(f)[10..20],
>> >>   should allow lazy evaluation.
>> >>
>> >> Feedback is welcome!
>> >>
>> >>
>> >> Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
>> >>
>> >> > Hey Daniel,
>> >> > I'm very sorry, but I didn't make any progress with this. I think I
>> was a
>> >> > bit over-motivated, but unfortunately I cannot spend more time on this
>> >> for
>> >> > various reasons.
>> >> > You can take this over. I'm glad to help out with testing and
>> feedback.
>> >> >
>> >> > Thanks
>> >> > Christoph
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <
>> >> ddekany@apache.org
>> >> >>:
>> >> >
>> >> >> Any progress in this? I think I will give it a try in the coming days
>> >> >> otherwise.
>> >> >>
>> >> >>
>> >> >> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
>> >> >>
>> >> >> > See my answers inline...
>> >> >> >
>> >> >> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
>> >> >> >
>> >> >> >> Thanks Daniel for your feedback. See my answers below
>> >> >> >>
>> >> >> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
>> >> >> ddekany@apache.org
>> >> >> >>>:
>> >> >> >>
>> >> >> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>> >> >> >>>
>> >> >> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
>> >> >> >>> ddekany@apache.org
>> >> >> >>> >>:
>> >> >> >>> >
>> >> >> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>> >> >> >>> >>
>> >> >> >>> >> > Hi,
>> >> >> >>> >> >
>> >> >> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <
>> >> c.rueger@synesty.com>
>> >> >> a
>> >> >> >>> >> écrit :
>> >> >> >>> >> >
>> >> >> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>> >> >> >>> >> ddekany@apache.org
>> >> >> >>> >> > :
>> >> >> >>> >> >
>> >> >> >>> >> > It's certainly tricky, but as far as I see possible (but
>> then,
>> >> who
>> >> >> >>> >> >
>> >> >> >>> >> > knows what will one find when actually working on it). It's
>> >> also a
>> >> >> >>> >> > feature missing a lot. It's especially missing for #list (I
>> >> know
>> >> >> that
>> >> >> >>> >> > you need it for something else), because if you filter the
>> >> items
>> >> >> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not
>> be
>> >> >> usable.
>> >> >> >>> >> >
>> >> >> >>> >> > Let me say that I disagree here.
>> >> >> >>> >> >
>> >> >> >>> >> > I do not think that closures are required for FreeMarker,
>> nor
>> >> that
>> >> >> >>> they
>> >> >> >>> >> are a good idea.
>> >> >> >>> >> >
>> >> >> >>> >> > If we add new features to the FreeMarker *tempate engine* I
>> >> would
>> >> >> >>> >> > rather we focus on multi-part macro body rather than an
>> >> advanced
>> >> >> >>> >> language feature like closures.
>> >> >> >>> >> >
>> >> >> >>> >> > You can add ?filter and ?map if you want, a simple
>> expression
>> >> as
>> >> >> >>> >> parameter should be enough.
>> >> >> >>> >>
>> >> >> >>> >> Yes, as I said, we certainly start with only allowing lambdas
>> in
>> >> >> >>> >> ?filter/?map, also certainly in ?contains.
>> >> >> >>> >>
>> >> >> >>> > Would be enough in my opinion and very useful.
>> >> >> >>> >
>> >> >> >>> > Is it possiblefor you to give some pointers to the code on how
>> >> this
>> >> >> could
>> >> >> >>> > be implemented? I would maybe like to wrap my head around this
>> a
>> >> >> little
>> >> >> >>> bit.
>> >> >> >>>
>> >> >> >>> Please feel yourself encouraged! (:
>> >> >> >>>
>> >> >> >>> > I started looking at seq_containsBI (
>> >> >> >>> >
>> >> >> >>>
>> >> >>
>> >>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
>> >> >> >>> )
>> >> >> >>> > and
>> >> >> >>> > and reverseBI (
>> >> >> >>> >
>> >> >> >>>
>> >> >>
>> >>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
>> >> >> >>> )
>> >> >> >>> > just to find something related (seq_containsBI checks
>> something)
>> >> and
>> >> >> >>> > reverseBI returns a new sequence.
>> >> >> >>> > What I haven't found is a function which takes an Expression
>> as a
>> >> >> >>> > parameter.
>> >> >> >>> > Is there something similar already or would that be a new
>> thing?
>> >> >> >>>
>> >> >> >>> It's a new thing in that it will be part of the expression syntax
>> >> >> >>> (even if for now we will only allow lambdas as the parameters of
>> a
>> >> few
>> >> >> >>> built-ins, so that we can get away without closures). So it's a
>> new
>> >> >> >>> Expression subclass, and has to be part of the parser (ftl.jj) as
>> >> >> >>> well.
>> >> >> >>
>> >> >> >> Hmm, that parser stuff is new for me, it'll take me some time to
>> get
>> >> >> into
>> >> >> >> it.
>> >> >> >
>> >> >> > So it's a JavaCC lexer+parser. With a few twists... sorry, but this
>> >> >> > code has history... :)
>> >> >> >
>> >> >> >>> As of lazy evaluation of parameters expressions, that's already
>> >> >> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and you
>> >> will
>> >> >> >>> see it's trivial to do, but the situation there is much simpler.
>> >> >> >>>
>> >> >> >>
>> >> >> >>
>> >> >> >>>
>> >> >> >>> In principle, a LambdaExpression should evaluate to a
>> >> >> >>> TemplateMethodModelEx, and then you pass that
>> TemplateMethodModelEx
>> >> to
>> >> >> >>> the called built-in or whatever it is. But with the approach of
>> >> >> >>> BuiltInsWithParseTimeParameters we can certainly even skip that,
>> and
>> >> >> >>> just bind to the LambdaExpression directly, add a LocalContext
>> that
>> >> >> >>> contains the lambda arguments, end evaluate the LambdaExpression
>> >> right
>> >> >> >>> there, in the built-in implementation. Or at least at a very
>> quick
>> >> >> >>> glance I think so.
>> >> >> >>>
>> >> >> >> Not sure I can follow completely but that hint with*
>> >> >> >> BuiltInsWithParseTimeParameters* got me started, but at the moment
>> >> I'm
>> >> >> >> stuck as I need to get more familiar with the internals of
>> >> Freemarker.
>> >> >> I am
>> >> >> >> also not sure I am on the same page regarding the syntax we are
>> >> aiming
>> >> >> for
>> >> >> >> and why I would need to extend the parser when there is something
>> >> like
>> >> >> >> BuiltInsWithParseTimeParameters....
>> >> >> >
>> >> >> > I assumed that the syntax will be similar to the Java lambda syntax
>> >> >> > (see below why), and that's of course needs lexer/parser changes.
>> >> >> >
>> >> >> >> Here is an example what I have in mind:
>> >> >> >> I started with the ?filter() builtin. I had a syntax like this in
>> >> mind:
>> >> >> >>
>> >> >> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
>> >> >> >> *Example 2: ["a","b","c"]?filter(element, element ==
>> >> someOtherVariable,
>> >> >> >> someOtherVariable="c")*
>> >> >> >
>> >> >> > Looking at the above one believe that the value of `element` is
>> passed
>> >> >> > in as first argument, and the the value of `element == "c"` as the
>> >> >> > second, but that's not the case. It's much better if it's visible
>> that
>> >> >> > you got some kind of anonymous function definition there without
>> >> >> > knowing about the "filter" built-in.
>> >> >> >
>> >> >> > Java already has a syntax for expressing this kind of thing, so it
>> >> >> > would be better to use that familiar syntax. As far as I know it
>> >> >> > doesn't conflict with ours (it kind of it does, but not fatally).
>> >> >> >
>> >> >> > Also, even if for now we only allow this in said built-ins, we
>> >> >> > shouldn't exclude the possibility of making this kind of expression
>> >> >> > accessible elsewhere as well. Then, on many places the parsed won't
>> >> >> > know what kind of value is expected (consider passing a lambda to
>> an
>> >> >> > user defined directive for example), so a syntax like above won't
>> >> >> > work.
>> >> >> >
>> >> >> >> Not sure if that's what you have in mind too, but to me it made
>> sense
>> >> >> with
>> >> >> >> regards to BuiltInsWithParseTimeParameters and I could start
>> without
>> >> >> >> touching parser stuff.
>> >> >> >
>> >> >> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
>> >> >> > bit...), as a call to a such built-in looks like a call to any
>> other
>> >> >> > built-in.
>> >> >> >
>> >> >> >> *1st argument 'element'* would just be the iterator variable
>> similar
>> >> to
>> >> >> >> <#list ["a","b","c"] as *element*>
>> >> >> >> 2nd argument is the filter lambda expression... aka our filter
>> >> condition
>> >> >> >> 3rd+n argument are optional parameters in case used in the lambda
>> >> >> expression
>> >> >> >
>> >> >> > #list is special, similarly as `for ... in ...` is in most
>> languages.
>> >> >> > It's not lambda-ish either; you can't write `as element/2` or such.
>> >> >> >
>> >> >> >> So at first I was looking how <#list> works and found
>> IteratorBlock,
>> >> and
>> >> >> >> though I could reuse it somehow.
>> >> >> >>
>> >> >> >> Here is some simple pseudo code I played around for the for
>> Example
>> >> 1:
>> >> >> >>
>> >> >> >> static class filter_BI extends BuiltInWithParseTimeParameters {
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>         TemplateModel _eval(Environment env) throws
>> >> TemplateException {
>> >> >> >>
>> >> >> >>             // sequence
>> >> >> >>
>> >> >> >>         TemplateModel targetValue = target.evalToNonMissing(env);
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>             List parameters = this.parameters;
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>             Expression iteratorAlias = (Expression)
>> >> parameters.get(0);
>> >> >> >>
>> >> >> >>             Expression conditionExpression = (Expression)
>> >> >> parameters.get(1);
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>             TemplateSequenceModel seq =  (TemplateSequenceModel)
>> >> >> target.eval
>> >> >> >> (env);
>> >> >> >>
>> >> >> >>             for (int i = 0; i < seq.size(); i++) {
>> >> >> >>
>> >> >> >>                TemplateModel cur = seq.get(i);
>> >> >> >>
>> >> >> >>
>> >> >> >>                // this is where I am stuck at the moment
>> >> >> >>
>> >> >> >>                // I basically want to evaluate conditionExpression
>> >> >> >> where iteratorAlias
>> >> >> >> is basically what I passed as 'element'
>> >> >> >>
>> >> >> >>                // I am not sure if or how LocalContext could come
>> >> into
>> >> >> play
>> >> >> >> here
>> >> >> >>
>> >> >> >>                // basically for each iteration I would assign the
>> >> >> current
>> >> >> >> loop element to a context variable with the name 'element'
>> >> >> >>
>> >> >> >>                // and then evaluate conditionExpression with that
>> >> >> context.
>> >> >> >>
>> >> >> >>                // if conditionExpression is "true" then I would
>> >> populate
>> >> >> >> add the current sequence element 'cur'
>> >> >> >>
>> >> >> >>                // to a new result-List.... and return that....
>> >> something
>> >> >> >>
>> >> >> >>                // I wanted to reuse IteratorBlock here somehow,
>> but
>> >> >> didn't
>> >> >> >> get it to work yet.
>> >> >> >>
>> >> >> >>                // maybe this is a stupid idea, or we just need
>> >> something
>> >> >> >> similar
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>             }
>> >> >> >>
>> >> >> >> }
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> Ok so far for my pseudo code....  Maybe you could give some more
>> >> >> pointers
>> >> >> >> based on that... in case this makes any sense ...
>> >> >> >
>> >> >> > For LocalContext, Environment has a pushLocalContext method. See
>> the
>> >> >> > calls to it, like in visit(TemplateElement[],
>> TemplateDirectiveModel,
>> >> >> > Map, List).
>> >> >> >
>> >> >> > To avoid running into more new things at once than necessary,
>> perhaps
>> >> >> > you should start with seq?seq_contains(lambda). The end result
>> should
>> >> >> > be like:
>> >> >> >
>> >> >> >   users?contains(u -> u.paying)
>> >> >> >
>> >> >> >>> Another similarity to BuiltInsWithParseTimeParameters is that we
>> >> won't
>> >> >> >>> allow separating the `?someBI` and `(arg)`. Like, with the
>> example
>> >> of
>> >> >> >>> `cond?then(1, 2)`, you aren't allowed to do this:
>> >> >> >>>
>> >> >> >>>   <#assign t=cond?then>
>> >> >> >>>   ${t(1, 2)}
>> >> >> >>>
>> >> >> >>> That maybe looks natural, but most other built-ins allow that. Of
>> >> >> >>> course we can't allow that for ?filter etc., because then we need
>> >> >> >>> closures again.
>> >> >> >>>
>> >> >> >>> >> Multi-part macro body is also planned. Means, I know it
>> >> definitely
>> >> >> >>> >> should be added, but who knows when that's done... I mean,
>> it's
>> >> like
>> >> >> >>> >> that for what, a decade? (: It's not even decided what it
>> exactly
>> >> >> >>> >> does, as there are many ways of approaching this. (I have my
>> own
>> >> >> idea
>> >> >> >>> >> about what the right compromise would be, but others has other
>> >> >> >>> >> ideas...)
>> >> >> >>> >>
>> >> >> >>> >> Filtering lists bothers me because the template language
>> should
>> >> be
>> >> >> >>> >> (and somewhat indeed is) specialized on listing things on
>> fancy
>> >> ways
>> >> >> >>> >> that used to come up when generating document-like output.
>> (If it
>> >> >> >>> >> doesn't do things like that, you might as well use a general
>> >> purpose
>> >> >> >>> >> language.) Thus, that filter is unsolved (filtering with #if
>> is
>> >> >> >>> >> verbose and spoils #sep etc.) bothers me a lot.
>> >> >> >>> >>
>> >> >> >>> >> BTW, ?filter and ?map is also especially handy in our case as
>> >> >> >>> >> FreeMarker doesn't support building new sequences (sequences
>> are
>> >> >> >>> >> immutable). Although it has sequence concatenation with `+`,
>> it's
>> >> >> not
>> >> >> >>> >> good for building a sequence one by one, unless the sequence
>> >> will be
>> >> >> >>> >> quite short.
>> >> >> >>> >>
>> >> >> >>> > Good point.
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> >>
>> >> >> >>> >> > Cheers,
>> >> >> >>> >> > -- Denis.
>> >> >> >>> >>
>> >> >> >>> >> --
>> >> >> >>> >> Thanks,
>> >> >> >>> >>  Daniel Dekany
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> Thanks,
>> >> >> >>>  Daniel Dekany
>> >> >> >>>
>> >> >> >>>
>> >> >> >> Thanks
>> >> >> >> Christoph
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >> --
>> >> >> Thanks,
>> >> >>  Daniel Dekany
>> >> >>
>> >> >>
>> >> >
>> >> > --
>> >> > Christoph Rüger, Geschäftsführer
>> >> > Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
>> >> > Programmieren - Automatisierung, Schnittstellen, Datenfeeds
>> >> >
>> >> > Xing: https://www.xing.com/profile/Christoph_Rueger2
>> >> > LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
>> >> >
>> >>
>> >> --
>> >> Thanks,
>> >>  Daniel Dekany
>> >>
>> >>
>> >
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
>

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

Am So., 24. Feb. 2019 um 19:40 Uhr schrieb Daniel Dekany <ddekany@apache.org
>:

> Sunday, February 24, 2019, 12:45:48 AM, Christoph Rüger wrote:
>
> > Thanks you very much. This looks great.
> > Our testsuite passes all freemarker related tests.
> >
> > Also some simple manual tests were successful e.g.
> > <#list row.cols?map(col -> col.title)?filter(ct -> ct?contains("name"))
> as
> coltitle >>${coltitle!}<#sep>,</#sep></#list>
> > <#list description?split(" ")?filter(word -> word == "test") as
> filteredWord>>${filteredWord}</#list>
> >
> > Regarding other use-cases for lazy-evaluation:
> >
> > Do you think a ?count (as terminal operation) without building up a list
> in
> > memory?
> > E.g.
> > Number of products < 1000: ${products?filter(it -> it.price <
> 1000)?count}
> >
> > (I tested ${products?filter(it -> it.price < 1000)?size} but I think this
> > builds up a list in memory which can be large)
>
> Good point! We don't need to introduce ?count for this though, simply
> ?size has to be made smarter.
>

Right, that's better. Let's use ?size.


>
> > Or what about other aggregations for numerical values like sum, avg, min,
> > max e.g.:
> > Cheapest price of products < 1000: ${products?filter(it -> it.price <
> > 1000)?map(it -> it.price)?min}
>
> We have ?min and ?max, they also need to be made smarter.


Ok. If it can be done efficiently then good. As ?min ?max are already there
I guess people would expect to use them...as I did.


> As of ?sum
> and ?acg... they are doable. But aggregates are mostly useful combined
> with "group by". Technically, we can do all that, but I'm not 100%
> sure if it's wise to do.

The more such operations the template
> language supports, the more people will tend to move such calculations
> from the Java code that builds the data-model to the template. But the
> good practice was always the opposite of that: only do presentation
> logic in templates, not "business" calculations. But then again, there
> are the situations when the template authors have little control over
> the data model (typically, when you make a report/mail template for
> some business application that was developed by another party), and
> then they are happy that they can get the work done in the template.
> So, it's not easy to decide which compromise is the better.
>

Yeah difficult decision.... We are such an application you mention where
the template is the only tool available.

Maybe groupBy and aggregates like ?sum, ?avg should not be part of it for
now. I think Grouping requires keeping data in memory... that's pretty
powerful but performance critical. Or have a way to enable/disable it with
configuration.

Speaking of configuration / customization: We have built something like
groupBy ourselves outside the template language (for a special table
object). We added some "safety" stuff to abort processing if the data grows
too large over a defined threshold. It would be good to be able to
customize this too.



> BTW, the kind of "group by" operation that doesn't aggregate anything,
> just splits the collection into smaller collections for each distinct
> "group by" value, like Collectors.groupBy in Java 8, would be often
> useful for presentation logic. The use case is that you get a List of
> rows, and let's say you have a Year column in there, and the List is
> ordered by that. Then you should render it so that you don't show the
> same year again and again. Like if it's a HTML table then you are
> supposed to use td.@rowspan in the year column. It's quite awkward to
> do such thing currently (try it and you will see...), despite that
> it's a common requirement and is clearly a presentation decision, not
> really "business logic". So, we could introduce a built-in for that:
> collection?splitBy(lambdaOrFunctionOrMethod). I would avoid calling it
> groupBy, since that has a different meaning in SQL (the aggregating
> groupBy veriant). So now rowspan by year would be like:
>
>   <table>
>     <tr>
>       <th>Year</th>
>       <th>Department</th>
>       <th>Income</th>
>     </tr>
>     <#list rows?groupBy(row -> row.year) as sameYearRows>
>       <#list sameYearRows as row>
>         <tr>
>           <#if row?isFirst>
>             <td rowspan="${sameYearRows?size}">${row.year}</td>
>           </#if>
>           <td>${row.department}</td>
>           <td>${row.income}</td>
>         </tr>
>       </#list>
>     <#list>
>   </table>
>
> Or if you have to print a new H2 and table for each year:
>
>   <#list rows?groupBy(row -> row.year) as sameYearRows>
>     <h2>${sameYearRows[0].year}<h2>
>     <table>
>       <tr>
>         <th>Department</th>
>         <th>Income</th>
>       </tr>
>       <#list sameYearRows as row>
>         <tr>
>           <td>${row.department}</td>
>           <td>${row.income}</td>
>         </tr>
>       </#list>
>     </table>
>   <#list>
>
>
Yes that can be handy. But the implementation should be careful to avoid
large in-memory structures for the "groups"
-> Does ?groupBy(row -> row.year) create a new List for each group? Or is
it kind of a "Views" on the original list?


> Lambdas happen to be handy with this as well. Actually, some long
> existing built-ins, like ?sortBy, ?contains, etc. could use them as
> well (a bit tricky for them technically due to backward compatibility
> requirements, but I belive I can work that around.)
>

Overall those lambdas allow some great new use cases with pretty concise
syntax. That is good. The ?filter and ?map built-ins are very cool already.
If some existing built-ins will be made "smarter" (as you said), then it
will be a great enhancement.

Speaking for us, performance and memory usage is always a concern. So it
would be good to keep an eye on avoiding large new in-memory structures.

My thoughts so far.
Thanks
Christoph


>
> > Didn't think this through, but just had the idea while testing.
> >
> > Anyway, good work!
> >
> >
> >
> >
> >
> >
> > Am Sa., 23. Feb. 2019 um 18:19 Uhr schrieb Daniel Dekany <
> ddekany@apache.org
> >>:
> >
> >> I have pushed an implementation of the restricted lambdas we were
> >> talking about. Guys, please review/test it.
> >>
> >> I call these "local lambdas" in the source code (but I'm open for
> >> suggestions), as the function they define can only be called in the
> >> same variable scope where the lambda was (as we have no closures in
> >> the template language, nor final variables). Also, they can only be
> >> used as the parameters of the built-ins that explicitly support them.
> >> As the subject shows, the main goal was just support filtering for
> >> #list, without a nested #if, as that breaks #sep, #else, it?hasNext,
> >> it?index, etc. (I will try to make more universal lambdas in FM3... I
> >> guess it would be way too tricky in FM2.)
> >>
> >> For now, I have only added two built-ins that support lambdas: ?filter
> >> and ?map. Any ideas what other such built-ins would be often useful in
> >> templates (with use case, if possible)?
> >>
> >> Examples of using ?filter and ?map:
> >>
> >>   <#list products?filter(it -> it.price < 1000) as product>
> >>     ${product.name}
> >>   </#list>
> >>
> >>   <#list products?map(it -> it.name) as name>
> >>     ${name}
> >>   </#list>
> >>
> >> Of course these built-ins aren't specific to #list, they can be used
> >> anywhere. Naturally, they can be chained as well:
> >>
> >>   <#assign chepProdNames = products
> >>      ?filter(it -> it.price < 1000)
> >>      ?map(it -> it.name)
> >>   >
> >>
> >> As a side note, ?filter and ?map also accepts FTL function-s and Java
> >> methods as its parameter, not only lambdas.
> >>
> >> A tricky aspect of this feature is lazy evaluation, in similar sense
> >> as Java 8 Stream intermediate operations are lazy. On most places,
> >> ?filer and ?map are eager, that is, they return a completed sequence
> >> of items. That's because our restricted lambdas only work correctly
> >> "locally". However, there are very common situations where it's clear
> >> that we can use lazy ("streaming") evaluation, as we know where the
> >> resulting stream of elements will be consumed:
> >>
> >> - One such case is when these kind of built-ins are chained. Like in
> >>   the last example, ?filter doesn't construct a List in memory,
> >>   instead the elements just flow through it (some is dropped, as it's
> >>   filtering), into ?map. Only the ?map at the end of the chain will
> >>   built a List eagerly. Some other built-ins also allow the left-hand
> >>   built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
> >>   is built anywhere, instead the elements flow through both ?filter
> >>   and ?map, and ?join just appends them to the StringBuilder where it
> >>   creates its results.
> >>
> >> - #list also enables lazy evaluation to its 1st parameter. So in the
> >>   #list examples above, yet again no List is built in memory.
> >>
> >> - There are other such cases, which I didn't implement yet. For
> >>   example the sequence slice operator, like, xs?filter(f)[10..20],
> >>   should allow lazy evaluation.
> >>
> >> Feedback is welcome!
> >>
> >>
> >> Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
> >>
> >> > Hey Daniel,
> >> > I'm very sorry, but I didn't make any progress with this. I think I
> was a
> >> > bit over-motivated, but unfortunately I cannot spend more time on this
> >> for
> >> > various reasons.
> >> > You can take this over. I'm glad to help out with testing and
> feedback.
> >> >
> >> > Thanks
> >> > Christoph
> >> >
> >> >
> >> >
> >> >
> >> > Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <
> >> ddekany@apache.org
> >> >>:
> >> >
> >> >> Any progress in this? I think I will give it a try in the coming days
> >> >> otherwise.
> >> >>
> >> >>
> >> >> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
> >> >>
> >> >> > See my answers inline...
> >> >> >
> >> >> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
> >> >> >
> >> >> >> Thanks Daniel for your feedback. See my answers below
> >> >> >>
> >> >> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
> >> >> ddekany@apache.org
> >> >> >>>:
> >> >> >>
> >> >> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
> >> >> >>>
> >> >> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
> >> >> >>> ddekany@apache.org
> >> >> >>> >>:
> >> >> >>> >
> >> >> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
> >> >> >>> >>
> >> >> >>> >> > Hi,
> >> >> >>> >> >
> >> >> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <
> >> c.rueger@synesty.com>
> >> >> a
> >> >> >>> >> écrit :
> >> >> >>> >> >
> >> >> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
> >> >> >>> >> ddekany@apache.org
> >> >> >>> >> > :
> >> >> >>> >> >
> >> >> >>> >> > It's certainly tricky, but as far as I see possible (but
> then,
> >> who
> >> >> >>> >> >
> >> >> >>> >> > knows what will one find when actually working on it). It's
> >> also a
> >> >> >>> >> > feature missing a lot. It's especially missing for #list (I
> >> know
> >> >> that
> >> >> >>> >> > you need it for something else), because if you filter the
> >> items
> >> >> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not
> be
> >> >> usable.
> >> >> >>> >> >
> >> >> >>> >> > Let me say that I disagree here.
> >> >> >>> >> >
> >> >> >>> >> > I do not think that closures are required for FreeMarker,
> nor
> >> that
> >> >> >>> they
> >> >> >>> >> are a good idea.
> >> >> >>> >> >
> >> >> >>> >> > If we add new features to the FreeMarker *tempate engine* I
> >> would
> >> >> >>> >> > rather we focus on multi-part macro body rather than an
> >> advanced
> >> >> >>> >> language feature like closures.
> >> >> >>> >> >
> >> >> >>> >> > You can add ?filter and ?map if you want, a simple
> expression
> >> as
> >> >> >>> >> parameter should be enough.
> >> >> >>> >>
> >> >> >>> >> Yes, as I said, we certainly start with only allowing lambdas
> in
> >> >> >>> >> ?filter/?map, also certainly in ?contains.
> >> >> >>> >>
> >> >> >>> > Would be enough in my opinion and very useful.
> >> >> >>> >
> >> >> >>> > Is it possiblefor you to give some pointers to the code on how
> >> this
> >> >> could
> >> >> >>> > be implemented? I would maybe like to wrap my head around this
> a
> >> >> little
> >> >> >>> bit.
> >> >> >>>
> >> >> >>> Please feel yourself encouraged! (:
> >> >> >>>
> >> >> >>> > I started looking at seq_containsBI (
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
> >> >> >>> )
> >> >> >>> > and
> >> >> >>> > and reverseBI (
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
> >> >> >>> )
> >> >> >>> > just to find something related (seq_containsBI checks
> something)
> >> and
> >> >> >>> > reverseBI returns a new sequence.
> >> >> >>> > What I haven't found is a function which takes an Expression
> as a
> >> >> >>> > parameter.
> >> >> >>> > Is there something similar already or would that be a new
> thing?
> >> >> >>>
> >> >> >>> It's a new thing in that it will be part of the expression syntax
> >> >> >>> (even if for now we will only allow lambdas as the parameters of
> a
> >> few
> >> >> >>> built-ins, so that we can get away without closures). So it's a
> new
> >> >> >>> Expression subclass, and has to be part of the parser (ftl.jj) as
> >> >> >>> well.
> >> >> >>
> >> >> >> Hmm, that parser stuff is new for me, it'll take me some time to
> get
> >> >> into
> >> >> >> it.
> >> >> >
> >> >> > So it's a JavaCC lexer+parser. With a few twists... sorry, but this
> >> >> > code has history... :)
> >> >> >
> >> >> >>> As of lazy evaluation of parameters expressions, that's already
> >> >> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and you
> >> will
> >> >> >>> see it's trivial to do, but the situation there is much simpler.
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >>>
> >> >> >>> In principle, a LambdaExpression should evaluate to a
> >> >> >>> TemplateMethodModelEx, and then you pass that
> TemplateMethodModelEx
> >> to
> >> >> >>> the called built-in or whatever it is. But with the approach of
> >> >> >>> BuiltInsWithParseTimeParameters we can certainly even skip that,
> and
> >> >> >>> just bind to the LambdaExpression directly, add a LocalContext
> that
> >> >> >>> contains the lambda arguments, end evaluate the LambdaExpression
> >> right
> >> >> >>> there, in the built-in implementation. Or at least at a very
> quick
> >> >> >>> glance I think so.
> >> >> >>>
> >> >> >> Not sure I can follow completely but that hint with*
> >> >> >> BuiltInsWithParseTimeParameters* got me started, but at the moment
> >> I'm
> >> >> >> stuck as I need to get more familiar with the internals of
> >> Freemarker.
> >> >> I am
> >> >> >> also not sure I am on the same page regarding the syntax we are
> >> aiming
> >> >> for
> >> >> >> and why I would need to extend the parser when there is something
> >> like
> >> >> >> BuiltInsWithParseTimeParameters....
> >> >> >
> >> >> > I assumed that the syntax will be similar to the Java lambda syntax
> >> >> > (see below why), and that's of course needs lexer/parser changes.
> >> >> >
> >> >> >> Here is an example what I have in mind:
> >> >> >> I started with the ?filter() builtin. I had a syntax like this in
> >> mind:
> >> >> >>
> >> >> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
> >> >> >> *Example 2: ["a","b","c"]?filter(element, element ==
> >> someOtherVariable,
> >> >> >> someOtherVariable="c")*
> >> >> >
> >> >> > Looking at the above one believe that the value of `element` is
> passed
> >> >> > in as first argument, and the the value of `element == "c"` as the
> >> >> > second, but that's not the case. It's much better if it's visible
> that
> >> >> > you got some kind of anonymous function definition there without
> >> >> > knowing about the "filter" built-in.
> >> >> >
> >> >> > Java already has a syntax for expressing this kind of thing, so it
> >> >> > would be better to use that familiar syntax. As far as I know it
> >> >> > doesn't conflict with ours (it kind of it does, but not fatally).
> >> >> >
> >> >> > Also, even if for now we only allow this in said built-ins, we
> >> >> > shouldn't exclude the possibility of making this kind of expression
> >> >> > accessible elsewhere as well. Then, on many places the parsed won't
> >> >> > know what kind of value is expected (consider passing a lambda to
> an
> >> >> > user defined directive for example), so a syntax like above won't
> >> >> > work.
> >> >> >
> >> >> >> Not sure if that's what you have in mind too, but to me it made
> sense
> >> >> with
> >> >> >> regards to BuiltInsWithParseTimeParameters and I could start
> without
> >> >> >> touching parser stuff.
> >> >> >
> >> >> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
> >> >> > bit...), as a call to a such built-in looks like a call to any
> other
> >> >> > built-in.
> >> >> >
> >> >> >> *1st argument 'element'* would just be the iterator variable
> similar
> >> to
> >> >> >> <#list ["a","b","c"] as *element*>
> >> >> >> 2nd argument is the filter lambda expression... aka our filter
> >> condition
> >> >> >> 3rd+n argument are optional parameters in case used in the lambda
> >> >> expression
> >> >> >
> >> >> > #list is special, similarly as `for ... in ...` is in most
> languages.
> >> >> > It's not lambda-ish either; you can't write `as element/2` or such.
> >> >> >
> >> >> >> So at first I was looking how <#list> works and found
> IteratorBlock,
> >> and
> >> >> >> though I could reuse it somehow.
> >> >> >>
> >> >> >> Here is some simple pseudo code I played around for the for
> Example
> >> 1:
> >> >> >>
> >> >> >> static class filter_BI extends BuiltInWithParseTimeParameters {
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>         TemplateModel _eval(Environment env) throws
> >> TemplateException {
> >> >> >>
> >> >> >>             // sequence
> >> >> >>
> >> >> >>         TemplateModel targetValue = target.evalToNonMissing(env);
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>             List parameters = this.parameters;
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>             Expression iteratorAlias = (Expression)
> >> parameters.get(0);
> >> >> >>
> >> >> >>             Expression conditionExpression = (Expression)
> >> >> parameters.get(1);
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>             TemplateSequenceModel seq =  (TemplateSequenceModel)
> >> >> target.eval
> >> >> >> (env);
> >> >> >>
> >> >> >>             for (int i = 0; i < seq.size(); i++) {
> >> >> >>
> >> >> >>                TemplateModel cur = seq.get(i);
> >> >> >>
> >> >> >>
> >> >> >>                // this is where I am stuck at the moment
> >> >> >>
> >> >> >>                // I basically want to evaluate conditionExpression
> >> >> >> where iteratorAlias
> >> >> >> is basically what I passed as 'element'
> >> >> >>
> >> >> >>                // I am not sure if or how LocalContext could come
> >> into
> >> >> play
> >> >> >> here
> >> >> >>
> >> >> >>                // basically for each iteration I would assign the
> >> >> current
> >> >> >> loop element to a context variable with the name 'element'
> >> >> >>
> >> >> >>                // and then evaluate conditionExpression with that
> >> >> context.
> >> >> >>
> >> >> >>                // if conditionExpression is "true" then I would
> >> populate
> >> >> >> add the current sequence element 'cur'
> >> >> >>
> >> >> >>                // to a new result-List.... and return that....
> >> something
> >> >> >>
> >> >> >>                // I wanted to reuse IteratorBlock here somehow,
> but
> >> >> didn't
> >> >> >> get it to work yet.
> >> >> >>
> >> >> >>                // maybe this is a stupid idea, or we just need
> >> something
> >> >> >> similar
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>             }
> >> >> >>
> >> >> >> }
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Ok so far for my pseudo code....  Maybe you could give some more
> >> >> pointers
> >> >> >> based on that... in case this makes any sense ...
> >> >> >
> >> >> > For LocalContext, Environment has a pushLocalContext method. See
> the
> >> >> > calls to it, like in visit(TemplateElement[],
> TemplateDirectiveModel,
> >> >> > Map, List).
> >> >> >
> >> >> > To avoid running into more new things at once than necessary,
> perhaps
> >> >> > you should start with seq?seq_contains(lambda). The end result
> should
> >> >> > be like:
> >> >> >
> >> >> >   users?contains(u -> u.paying)
> >> >> >
> >> >> >>> Another similarity to BuiltInsWithParseTimeParameters is that we
> >> won't
> >> >> >>> allow separating the `?someBI` and `(arg)`. Like, with the
> example
> >> of
> >> >> >>> `cond?then(1, 2)`, you aren't allowed to do this:
> >> >> >>>
> >> >> >>>   <#assign t=cond?then>
> >> >> >>>   ${t(1, 2)}
> >> >> >>>
> >> >> >>> That maybe looks natural, but most other built-ins allow that. Of
> >> >> >>> course we can't allow that for ?filter etc., because then we need
> >> >> >>> closures again.
> >> >> >>>
> >> >> >>> >> Multi-part macro body is also planned. Means, I know it
> >> definitely
> >> >> >>> >> should be added, but who knows when that's done... I mean,
> it's
> >> like
> >> >> >>> >> that for what, a decade? (: It's not even decided what it
> exactly
> >> >> >>> >> does, as there are many ways of approaching this. (I have my
> own
> >> >> idea
> >> >> >>> >> about what the right compromise would be, but others has other
> >> >> >>> >> ideas...)
> >> >> >>> >>
> >> >> >>> >> Filtering lists bothers me because the template language
> should
> >> be
> >> >> >>> >> (and somewhat indeed is) specialized on listing things on
> fancy
> >> ways
> >> >> >>> >> that used to come up when generating document-like output.
> (If it
> >> >> >>> >> doesn't do things like that, you might as well use a general
> >> purpose
> >> >> >>> >> language.) Thus, that filter is unsolved (filtering with #if
> is
> >> >> >>> >> verbose and spoils #sep etc.) bothers me a lot.
> >> >> >>> >>
> >> >> >>> >> BTW, ?filter and ?map is also especially handy in our case as
> >> >> >>> >> FreeMarker doesn't support building new sequences (sequences
> are
> >> >> >>> >> immutable). Although it has sequence concatenation with `+`,
> it's
> >> >> not
> >> >> >>> >> good for building a sequence one by one, unless the sequence
> >> will be
> >> >> >>> >> quite short.
> >> >> >>> >>
> >> >> >>> > Good point.
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >>
> >> >> >>> >> > Cheers,
> >> >> >>> >> > -- Denis.
> >> >> >>> >>
> >> >> >>> >> --
> >> >> >>> >> Thanks,
> >> >> >>> >>  Daniel Dekany
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >
> >> >> >>>
> >> >> >>> --
> >> >> >>> Thanks,
> >> >> >>>  Daniel Dekany
> >> >> >>>
> >> >> >>>
> >> >> >> Thanks
> >> >> >> Christoph
> >> >> >>
> >> >> >
> >> >>
> >> >> --
> >> >> Thanks,
> >> >>  Daniel Dekany
> >> >>
> >> >>
> >> >
> >> > --
> >> > Christoph Rüger, Geschäftsführer
> >> > Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
> >> > Programmieren - Automatisierung, Schnittstellen, Datenfeeds
> >> >
> >> > Xing: https://www.xing.com/profile/Christoph_Rueger2
> >> > LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
> >> >
> >>
> >> --
> >> Thanks,
> >>  Daniel Dekany
> >>
> >>
> >
>
> --
> Thanks,
>  Daniel Dekany
>
>

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 
5596493Internet: https://synesty.com <https://synesty.com>
Informationen 
zum Datenschutz: https://synesty.com/datenschutz 
<https://synesty.com/datenschutz>

Geschäftsführer: Christoph Rüger

Unternehmenssitz: Jena
Handelsregister B beim Amtsgericht: Jena

Handelsregister-Nummer: HRB 508766
Ust-IdNr.: DE287564982

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

Sunday, February 24, 2019, 12:45:48 AM, Christoph Rüger wrote:

> Thanks you very much. This looks great.
> Our testsuite passes all freemarker related tests.
>
> Also some simple manual tests were successful e.g.
> <#list row.cols?map(col -> col.title)?filter(ct -> ct?contains("name")) as
coltitle >>${coltitle!}<#sep>,</#sep></#list>
> <#list description?split(" ")?filter(word -> word == "test") as
filteredWord>>${filteredWord}</#list>
>
> Regarding other use-cases for lazy-evaluation:
>
> Do you think a ?count (as terminal operation) without building up a list in
> memory?
> E.g.
> Number of products < 1000: ${products?filter(it -> it.price < 1000)?count}
>
> (I tested ${products?filter(it -> it.price < 1000)?size} but I think this
> builds up a list in memory which can be large)

Good point! We don't need to introduce ?count for this though, simply
?size has to be made smarter.

> Or what about other aggregations for numerical values like sum, avg, min,
> max e.g.:
> Cheapest price of products < 1000: ${products?filter(it -> it.price <
> 1000)?map(it -> it.price)?min}

We have ?min and ?max, they also need to be made smarter. As of ?sum
and ?acg... they are doable. But aggregates are mostly useful combined
with "group by". Technically, we can do all that, but I'm not 100%
sure if it's wise to do. The more such operations the template
language supports, the more people will tend to move such calculations
from the Java code that builds the data-model to the template. But the
good practice was always the opposite of that: only do presentation
logic in templates, not "business" calculations. But then again, there
are the situations when the template authors have little control over
the data model (typically, when you make a report/mail template for
some business application that was developed by another party), and
then they are happy that they can get the work done in the template.
So, it's not easy to decide which compromise is the better.

BTW, the kind of "group by" operation that doesn't aggregate anything,
just splits the collection into smaller collections for each distinct
"group by" value, like Collectors.groupBy in Java 8, would be often
useful for presentation logic. The use case is that you get a List of
rows, and let's say you have a Year column in there, and the List is
ordered by that. Then you should render it so that you don't show the
same year again and again. Like if it's a HTML table then you are
supposed to use td.@rowspan in the year column. It's quite awkward to
do such thing currently (try it and you will see...), despite that
it's a common requirement and is clearly a presentation decision, not
really "business logic". So, we could introduce a built-in for that:
collection?splitBy(lambdaOrFunctionOrMethod). I would avoid calling it
groupBy, since that has a different meaning in SQL (the aggregating
groupBy veriant). So now rowspan by year would be like:

  <table>
    <tr>
      <th>Year</th>
      <th>Department</th>
      <th>Income</th>
    </tr>
    <#list rows?groupBy(row -> row.year) as sameYearRows>
      <#list sameYearRows as row>
        <tr>
          <#if row?isFirst>
            <td rowspan="${sameYearRows?size}">${row.year}</td>
          </#if>
          <td>${row.department}</td>
          <td>${row.income}</td>
        </tr>
      </#list>
    <#list>
  </table>

Or if you have to print a new H2 and table for each year:

  <#list rows?groupBy(row -> row.year) as sameYearRows>
    <h2>${sameYearRows[0].year}<h2>
    <table>
      <tr>
        <th>Department</th>
        <th>Income</th>
      </tr>
      <#list sameYearRows as row>
        <tr>
          <td>${row.department}</td>
          <td>${row.income}</td>
        </tr>
      </#list>
    </table>
  <#list>

Lambdas happen to be handy with this as well. Actually, some long
existing built-ins, like ?sortBy, ?contains, etc. could use them as
well (a bit tricky for them technically due to backward compatibility
requirements, but I belive I can work that around.)

> Didn't think this through, but just had the idea while testing.
>
> Anyway, good work!
>
>
>
>
>
>
> Am Sa., 23. Feb. 2019 um 18:19 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>:
>
>> I have pushed an implementation of the restricted lambdas we were
>> talking about. Guys, please review/test it.
>>
>> I call these "local lambdas" in the source code (but I'm open for
>> suggestions), as the function they define can only be called in the
>> same variable scope where the lambda was (as we have no closures in
>> the template language, nor final variables). Also, they can only be
>> used as the parameters of the built-ins that explicitly support them.
>> As the subject shows, the main goal was just support filtering for
>> #list, without a nested #if, as that breaks #sep, #else, it?hasNext,
>> it?index, etc. (I will try to make more universal lambdas in FM3... I
>> guess it would be way too tricky in FM2.)
>>
>> For now, I have only added two built-ins that support lambdas: ?filter
>> and ?map. Any ideas what other such built-ins would be often useful in
>> templates (with use case, if possible)?
>>
>> Examples of using ?filter and ?map:
>>
>>   <#list products?filter(it -> it.price < 1000) as product>
>>     ${product.name}
>>   </#list>
>>
>>   <#list products?map(it -> it.name) as name>
>>     ${name}
>>   </#list>
>>
>> Of course these built-ins aren't specific to #list, they can be used
>> anywhere. Naturally, they can be chained as well:
>>
>>   <#assign chepProdNames = products
>>      ?filter(it -> it.price < 1000)
>>      ?map(it -> it.name)
>>   >
>>
>> As a side note, ?filter and ?map also accepts FTL function-s and Java
>> methods as its parameter, not only lambdas.
>>
>> A tricky aspect of this feature is lazy evaluation, in similar sense
>> as Java 8 Stream intermediate operations are lazy. On most places,
>> ?filer and ?map are eager, that is, they return a completed sequence
>> of items. That's because our restricted lambdas only work correctly
>> "locally". However, there are very common situations where it's clear
>> that we can use lazy ("streaming") evaluation, as we know where the
>> resulting stream of elements will be consumed:
>>
>> - One such case is when these kind of built-ins are chained. Like in
>>   the last example, ?filter doesn't construct a List in memory,
>>   instead the elements just flow through it (some is dropped, as it's
>>   filtering), into ?map. Only the ?map at the end of the chain will
>>   built a List eagerly. Some other built-ins also allow the left-hand
>>   built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
>>   is built anywhere, instead the elements flow through both ?filter
>>   and ?map, and ?join just appends them to the StringBuilder where it
>>   creates its results.
>>
>> - #list also enables lazy evaluation to its 1st parameter. So in the
>>   #list examples above, yet again no List is built in memory.
>>
>> - There are other such cases, which I didn't implement yet. For
>>   example the sequence slice operator, like, xs?filter(f)[10..20],
>>   should allow lazy evaluation.
>>
>> Feedback is welcome!
>>
>>
>> Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
>>
>> > Hey Daniel,
>> > I'm very sorry, but I didn't make any progress with this. I think I was a
>> > bit over-motivated, but unfortunately I cannot spend more time on this
>> for
>> > various reasons.
>> > You can take this over. I'm glad to help out with testing and feedback.
>> >
>> > Thanks
>> > Christoph
>> >
>> >
>> >
>> >
>> > Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <
>> ddekany@apache.org
>> >>:
>> >
>> >> Any progress in this? I think I will give it a try in the coming days
>> >> otherwise.
>> >>
>> >>
>> >> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
>> >>
>> >> > See my answers inline...
>> >> >
>> >> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
>> >> >
>> >> >> Thanks Daniel for your feedback. See my answers below
>> >> >>
>> >> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
>> >> ddekany@apache.org
>> >> >>>:
>> >> >>
>> >> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>> >> >>>
>> >> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
>> >> >>> ddekany@apache.org
>> >> >>> >>:
>> >> >>> >
>> >> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>> >> >>> >>
>> >> >>> >> > Hi,
>> >> >>> >> >
>> >> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <
>> c.rueger@synesty.com>
>> >> a
>> >> >>> >> écrit :
>> >> >>> >> >
>> >> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>> >> >>> >> ddekany@apache.org
>> >> >>> >> > :
>> >> >>> >> >
>> >> >>> >> > It's certainly tricky, but as far as I see possible (but then,
>> who
>> >> >>> >> >
>> >> >>> >> > knows what will one find when actually working on it). It's
>> also a
>> >> >>> >> > feature missing a lot. It's especially missing for #list (I
>> know
>> >> that
>> >> >>> >> > you need it for something else), because if you filter the
>> items
>> >> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be
>> >> usable.
>> >> >>> >> >
>> >> >>> >> > Let me say that I disagree here.
>> >> >>> >> >
>> >> >>> >> > I do not think that closures are required for FreeMarker, nor
>> that
>> >> >>> they
>> >> >>> >> are a good idea.
>> >> >>> >> >
>> >> >>> >> > If we add new features to the FreeMarker *tempate engine* I
>> would
>> >> >>> >> > rather we focus on multi-part macro body rather than an
>> advanced
>> >> >>> >> language feature like closures.
>> >> >>> >> >
>> >> >>> >> > You can add ?filter and ?map if you want, a simple expression
>> as
>> >> >>> >> parameter should be enough.
>> >> >>> >>
>> >> >>> >> Yes, as I said, we certainly start with only allowing lambdas in
>> >> >>> >> ?filter/?map, also certainly in ?contains.
>> >> >>> >>
>> >> >>> > Would be enough in my opinion and very useful.
>> >> >>> >
>> >> >>> > Is it possiblefor you to give some pointers to the code on how
>> this
>> >> could
>> >> >>> > be implemented? I would maybe like to wrap my head around this a
>> >> little
>> >> >>> bit.
>> >> >>>
>> >> >>> Please feel yourself encouraged! (:
>> >> >>>
>> >> >>> > I started looking at seq_containsBI (
>> >> >>> >
>> >> >>>
>> >>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
>> >> >>> )
>> >> >>> > and
>> >> >>> > and reverseBI (
>> >> >>> >
>> >> >>>
>> >>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
>> >> >>> )
>> >> >>> > just to find something related (seq_containsBI checks something)
>> and
>> >> >>> > reverseBI returns a new sequence.
>> >> >>> > What I haven't found is a function which takes an Expression as a
>> >> >>> > parameter.
>> >> >>> > Is there something similar already or would that be a new thing?
>> >> >>>
>> >> >>> It's a new thing in that it will be part of the expression syntax
>> >> >>> (even if for now we will only allow lambdas as the parameters of a
>> few
>> >> >>> built-ins, so that we can get away without closures). So it's a new
>> >> >>> Expression subclass, and has to be part of the parser (ftl.jj) as
>> >> >>> well.
>> >> >>
>> >> >> Hmm, that parser stuff is new for me, it'll take me some time to get
>> >> into
>> >> >> it.
>> >> >
>> >> > So it's a JavaCC lexer+parser. With a few twists... sorry, but this
>> >> > code has history... :)
>> >> >
>> >> >>> As of lazy evaluation of parameters expressions, that's already
>> >> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and you
>> will
>> >> >>> see it's trivial to do, but the situation there is much simpler.
>> >> >>>
>> >> >>
>> >> >>
>> >> >>>
>> >> >>> In principle, a LambdaExpression should evaluate to a
>> >> >>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx
>> to
>> >> >>> the called built-in or whatever it is. But with the approach of
>> >> >>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
>> >> >>> just bind to the LambdaExpression directly, add a LocalContext that
>> >> >>> contains the lambda arguments, end evaluate the LambdaExpression
>> right
>> >> >>> there, in the built-in implementation. Or at least at a very quick
>> >> >>> glance I think so.
>> >> >>>
>> >> >> Not sure I can follow completely but that hint with*
>> >> >> BuiltInsWithParseTimeParameters* got me started, but at the moment
>> I'm
>> >> >> stuck as I need to get more familiar with the internals of
>> Freemarker.
>> >> I am
>> >> >> also not sure I am on the same page regarding the syntax we are
>> aiming
>> >> for
>> >> >> and why I would need to extend the parser when there is something
>> like
>> >> >> BuiltInsWithParseTimeParameters....
>> >> >
>> >> > I assumed that the syntax will be similar to the Java lambda syntax
>> >> > (see below why), and that's of course needs lexer/parser changes.
>> >> >
>> >> >> Here is an example what I have in mind:
>> >> >> I started with the ?filter() builtin. I had a syntax like this in
>> mind:
>> >> >>
>> >> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
>> >> >> *Example 2: ["a","b","c"]?filter(element, element ==
>> someOtherVariable,
>> >> >> someOtherVariable="c")*
>> >> >
>> >> > Looking at the above one believe that the value of `element` is passed
>> >> > in as first argument, and the the value of `element == "c"` as the
>> >> > second, but that's not the case. It's much better if it's visible that
>> >> > you got some kind of anonymous function definition there without
>> >> > knowing about the "filter" built-in.
>> >> >
>> >> > Java already has a syntax for expressing this kind of thing, so it
>> >> > would be better to use that familiar syntax. As far as I know it
>> >> > doesn't conflict with ours (it kind of it does, but not fatally).
>> >> >
>> >> > Also, even if for now we only allow this in said built-ins, we
>> >> > shouldn't exclude the possibility of making this kind of expression
>> >> > accessible elsewhere as well. Then, on many places the parsed won't
>> >> > know what kind of value is expected (consider passing a lambda to an
>> >> > user defined directive for example), so a syntax like above won't
>> >> > work.
>> >> >
>> >> >> Not sure if that's what you have in mind too, but to me it made sense
>> >> with
>> >> >> regards to BuiltInsWithParseTimeParameters and I could start without
>> >> >> touching parser stuff.
>> >> >
>> >> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
>> >> > bit...), as a call to a such built-in looks like a call to any other
>> >> > built-in.
>> >> >
>> >> >> *1st argument 'element'* would just be the iterator variable similar
>> to
>> >> >> <#list ["a","b","c"] as *element*>
>> >> >> 2nd argument is the filter lambda expression... aka our filter
>> condition
>> >> >> 3rd+n argument are optional parameters in case used in the lambda
>> >> expression
>> >> >
>> >> > #list is special, similarly as `for ... in ...` is in most languages.
>> >> > It's not lambda-ish either; you can't write `as element/2` or such.
>> >> >
>> >> >> So at first I was looking how <#list> works and found IteratorBlock,
>> and
>> >> >> though I could reuse it somehow.
>> >> >>
>> >> >> Here is some simple pseudo code I played around for the for Example
>> 1:
>> >> >>
>> >> >> static class filter_BI extends BuiltInWithParseTimeParameters {
>> >> >>
>> >> >>
>> >> >>
>> >> >>         TemplateModel _eval(Environment env) throws
>> TemplateException {
>> >> >>
>> >> >>             // sequence
>> >> >>
>> >> >>         TemplateModel targetValue = target.evalToNonMissing(env);
>> >> >>
>> >> >>
>> >> >>
>> >> >>             List parameters = this.parameters;
>> >> >>
>> >> >>
>> >> >>
>> >> >>             Expression iteratorAlias = (Expression)
>> parameters.get(0);
>> >> >>
>> >> >>             Expression conditionExpression = (Expression)
>> >> parameters.get(1);
>> >> >>
>> >> >>
>> >> >>
>> >> >>             TemplateSequenceModel seq =  (TemplateSequenceModel)
>> >> target.eval
>> >> >> (env);
>> >> >>
>> >> >>             for (int i = 0; i < seq.size(); i++) {
>> >> >>
>> >> >>                TemplateModel cur = seq.get(i);
>> >> >>
>> >> >>
>> >> >>                // this is where I am stuck at the moment
>> >> >>
>> >> >>                // I basically want to evaluate conditionExpression
>> >> >> where iteratorAlias
>> >> >> is basically what I passed as 'element'
>> >> >>
>> >> >>                // I am not sure if or how LocalContext could come
>> into
>> >> play
>> >> >> here
>> >> >>
>> >> >>                // basically for each iteration I would assign the
>> >> current
>> >> >> loop element to a context variable with the name 'element'
>> >> >>
>> >> >>                // and then evaluate conditionExpression with that
>> >> context.
>> >> >>
>> >> >>                // if conditionExpression is "true" then I would
>> populate
>> >> >> add the current sequence element 'cur'
>> >> >>
>> >> >>                // to a new result-List.... and return that....
>> something
>> >> >>
>> >> >>                // I wanted to reuse IteratorBlock here somehow, but
>> >> didn't
>> >> >> get it to work yet.
>> >> >>
>> >> >>                // maybe this is a stupid idea, or we just need
>> something
>> >> >> similar
>> >> >>
>> >> >>
>> >> >>
>> >> >>             }
>> >> >>
>> >> >> }
>> >> >>
>> >> >>
>> >> >>
>> >> >> Ok so far for my pseudo code....  Maybe you could give some more
>> >> pointers
>> >> >> based on that... in case this makes any sense ...
>> >> >
>> >> > For LocalContext, Environment has a pushLocalContext method. See the
>> >> > calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
>> >> > Map, List).
>> >> >
>> >> > To avoid running into more new things at once than necessary, perhaps
>> >> > you should start with seq?seq_contains(lambda). The end result should
>> >> > be like:
>> >> >
>> >> >   users?contains(u -> u.paying)
>> >> >
>> >> >>> Another similarity to BuiltInsWithParseTimeParameters is that we
>> won't
>> >> >>> allow separating the `?someBI` and `(arg)`. Like, with the example
>> of
>> >> >>> `cond?then(1, 2)`, you aren't allowed to do this:
>> >> >>>
>> >> >>>   <#assign t=cond?then>
>> >> >>>   ${t(1, 2)}
>> >> >>>
>> >> >>> That maybe looks natural, but most other built-ins allow that. Of
>> >> >>> course we can't allow that for ?filter etc., because then we need
>> >> >>> closures again.
>> >> >>>
>> >> >>> >> Multi-part macro body is also planned. Means, I know it
>> definitely
>> >> >>> >> should be added, but who knows when that's done... I mean, it's
>> like
>> >> >>> >> that for what, a decade? (: It's not even decided what it exactly
>> >> >>> >> does, as there are many ways of approaching this. (I have my own
>> >> idea
>> >> >>> >> about what the right compromise would be, but others has other
>> >> >>> >> ideas...)
>> >> >>> >>
>> >> >>> >> Filtering lists bothers me because the template language should
>> be
>> >> >>> >> (and somewhat indeed is) specialized on listing things on fancy
>> ways
>> >> >>> >> that used to come up when generating document-like output. (If it
>> >> >>> >> doesn't do things like that, you might as well use a general
>> purpose
>> >> >>> >> language.) Thus, that filter is unsolved (filtering with #if is
>> >> >>> >> verbose and spoils #sep etc.) bothers me a lot.
>> >> >>> >>
>> >> >>> >> BTW, ?filter and ?map is also especially handy in our case as
>> >> >>> >> FreeMarker doesn't support building new sequences (sequences are
>> >> >>> >> immutable). Although it has sequence concatenation with `+`, it's
>> >> not
>> >> >>> >> good for building a sequence one by one, unless the sequence
>> will be
>> >> >>> >> quite short.
>> >> >>> >>
>> >> >>> > Good point.
>> >> >>> >
>> >> >>> >
>> >> >>> >>
>> >> >>> >> > Cheers,
>> >> >>> >> > -- Denis.
>> >> >>> >>
>> >> >>> >> --
>> >> >>> >> Thanks,
>> >> >>> >>  Daniel Dekany
>> >> >>> >>
>> >> >>> >>
>> >> >>> >
>> >> >>>
>> >> >>> --
>> >> >>> Thanks,
>> >> >>>  Daniel Dekany
>> >> >>>
>> >> >>>
>> >> >> Thanks
>> >> >> Christoph
>> >> >>
>> >> >
>> >>
>> >> --
>> >> Thanks,
>> >>  Daniel Dekany
>> >>
>> >>
>> >
>> > --
>> > Christoph Rüger, Geschäftsführer
>> > Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
>> > Programmieren - Automatisierung, Schnittstellen, Datenfeeds
>> >
>> > Xing: https://www.xing.com/profile/Christoph_Rueger2
>> > LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
>> >
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
>

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

Thanks you very much. This looks great.
Our testsuite passes all freemarker related tests.

Also some simple manual tests were successful e.g.
<#list row.cols?map(col -> col.title)?filter(ct -> ct?contains("name")) as
coltitle >${coltitle!}<#sep>,</#sep></#list>
<#list description?split(" ")?filter(word -> word == "test") as
filteredWord>${filteredWord}</#list>

Regarding other use-cases for lazy-evaluation:

Do you think a ?count (as terminal operation) without building up a list in
memory?
E.g.
Number of products < 1000: ${products?filter(it -> it.price < 1000)?count}

(I tested ${products?filter(it -> it.price < 1000)?size} but I think this
builds up a list in memory which can be large)

Or what about other aggregations for numerical values like sum, avg, min,
max e.g.:
Cheapest price of products < 1000: ${products?filter(it -> it.price <
1000)?map(it -> it.price)?min}

Didn't think this through, but just had the idea while testing.

Anyway, good work!






Am Sa., 23. Feb. 2019 um 18:19 Uhr schrieb Daniel Dekany <ddekany@apache.org
>:

> I have pushed an implementation of the restricted lambdas we were
> talking about. Guys, please review/test it.
>
> I call these "local lambdas" in the source code (but I'm open for
> suggestions), as the function they define can only be called in the
> same variable scope where the lambda was (as we have no closures in
> the template language, nor final variables). Also, they can only be
> used as the parameters of the built-ins that explicitly support them.
> As the subject shows, the main goal was just support filtering for
> #list, without a nested #if, as that breaks #sep, #else, it?hasNext,
> it?index, etc. (I will try to make more universal lambdas in FM3... I
> guess it would be way too tricky in FM2.)
>
> For now, I have only added two built-ins that support lambdas: ?filter
> and ?map. Any ideas what other such built-ins would be often useful in
> templates (with use case, if possible)?
>
> Examples of using ?filter and ?map:
>
>   <#list products?filter(it -> it.price < 1000) as product>
>     ${product.name}
>   </#list>
>
>   <#list products?map(it -> it.name) as name>
>     ${name}
>   </#list>
>
> Of course these built-ins aren't specific to #list, they can be used
> anywhere. Naturally, they can be chained as well:
>
>   <#assign chepProdNames = products
>      ?filter(it -> it.price < 1000)
>      ?map(it -> it.name)
>   >
>
> As a side note, ?filter and ?map also accepts FTL function-s and Java
> methods as its parameter, not only lambdas.
>
> A tricky aspect of this feature is lazy evaluation, in similar sense
> as Java 8 Stream intermediate operations are lazy. On most places,
> ?filer and ?map are eager, that is, they return a completed sequence
> of items. That's because our restricted lambdas only work correctly
> "locally". However, there are very common situations where it's clear
> that we can use lazy ("streaming") evaluation, as we know where the
> resulting stream of elements will be consumed:
>
> - One such case is when these kind of built-ins are chained. Like in
>   the last example, ?filter doesn't construct a List in memory,
>   instead the elements just flow through it (some is dropped, as it's
>   filtering), into ?map. Only the ?map at the end of the chain will
>   built a List eagerly. Some other built-ins also allow the left-hand
>   built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
>   is built anywhere, instead the elements flow through both ?filter
>   and ?map, and ?join just appends them to the StringBuilder where it
>   creates its results.
>
> - #list also enables lazy evaluation to its 1st parameter. So in the
>   #list examples above, yet again no List is built in memory.
>
> - There are other such cases, which I didn't implement yet. For
>   example the sequence slice operator, like, xs?filter(f)[10..20],
>   should allow lazy evaluation.
>
> Feedback is welcome!
>
>
> Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
>
> > Hey Daniel,
> > I'm very sorry, but I didn't make any progress with this. I think I was a
> > bit over-motivated, but unfortunately I cannot spend more time on this
> for
> > various reasons.
> > You can take this over. I'm glad to help out with testing and feedback.
> >
> > Thanks
> > Christoph
> >
> >
> >
> >
> > Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <
> ddekany@apache.org
> >>:
> >
> >> Any progress in this? I think I will give it a try in the coming days
> >> otherwise.
> >>
> >>
> >> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
> >>
> >> > See my answers inline...
> >> >
> >> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
> >> >
> >> >> Thanks Daniel for your feedback. See my answers below
> >> >>
> >> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
> >> ddekany@apache.org
> >> >>>:
> >> >>
> >> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
> >> >>>
> >> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
> >> >>> ddekany@apache.org
> >> >>> >>:
> >> >>> >
> >> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
> >> >>> >>
> >> >>> >> > Hi,
> >> >>> >> >
> >> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <
> c.rueger@synesty.com>
> >> a
> >> >>> >> écrit :
> >> >>> >> >
> >> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
> >> >>> >> ddekany@apache.org
> >> >>> >> > :
> >> >>> >> >
> >> >>> >> > It's certainly tricky, but as far as I see possible (but then,
> who
> >> >>> >> >
> >> >>> >> > knows what will one find when actually working on it). It's
> also a
> >> >>> >> > feature missing a lot. It's especially missing for #list (I
> know
> >> that
> >> >>> >> > you need it for something else), because if you filter the
> items
> >> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be
> >> usable.
> >> >>> >> >
> >> >>> >> > Let me say that I disagree here.
> >> >>> >> >
> >> >>> >> > I do not think that closures are required for FreeMarker, nor
> that
> >> >>> they
> >> >>> >> are a good idea.
> >> >>> >> >
> >> >>> >> > If we add new features to the FreeMarker *tempate engine* I
> would
> >> >>> >> > rather we focus on multi-part macro body rather than an
> advanced
> >> >>> >> language feature like closures.
> >> >>> >> >
> >> >>> >> > You can add ?filter and ?map if you want, a simple expression
> as
> >> >>> >> parameter should be enough.
> >> >>> >>
> >> >>> >> Yes, as I said, we certainly start with only allowing lambdas in
> >> >>> >> ?filter/?map, also certainly in ?contains.
> >> >>> >>
> >> >>> > Would be enough in my opinion and very useful.
> >> >>> >
> >> >>> > Is it possiblefor you to give some pointers to the code on how
> this
> >> could
> >> >>> > be implemented? I would maybe like to wrap my head around this a
> >> little
> >> >>> bit.
> >> >>>
> >> >>> Please feel yourself encouraged! (:
> >> >>>
> >> >>> > I started looking at seq_containsBI (
> >> >>> >
> >> >>>
> >>
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
> >> >>> )
> >> >>> > and
> >> >>> > and reverseBI (
> >> >>> >
> >> >>>
> >>
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
> >> >>> )
> >> >>> > just to find something related (seq_containsBI checks something)
> and
> >> >>> > reverseBI returns a new sequence.
> >> >>> > What I haven't found is a function which takes an Expression as a
> >> >>> > parameter.
> >> >>> > Is there something similar already or would that be a new thing?
> >> >>>
> >> >>> It's a new thing in that it will be part of the expression syntax
> >> >>> (even if for now we will only allow lambdas as the parameters of a
> few
> >> >>> built-ins, so that we can get away without closures). So it's a new
> >> >>> Expression subclass, and has to be part of the parser (ftl.jj) as
> >> >>> well.
> >> >>
> >> >> Hmm, that parser stuff is new for me, it'll take me some time to get
> >> into
> >> >> it.
> >> >
> >> > So it's a JavaCC lexer+parser. With a few twists... sorry, but this
> >> > code has history... :)
> >> >
> >> >>> As of lazy evaluation of parameters expressions, that's already
> >> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and you
> will
> >> >>> see it's trivial to do, but the situation there is much simpler.
> >> >>>
> >> >>
> >> >>
> >> >>>
> >> >>> In principle, a LambdaExpression should evaluate to a
> >> >>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx
> to
> >> >>> the called built-in or whatever it is. But with the approach of
> >> >>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
> >> >>> just bind to the LambdaExpression directly, add a LocalContext that
> >> >>> contains the lambda arguments, end evaluate the LambdaExpression
> right
> >> >>> there, in the built-in implementation. Or at least at a very quick
> >> >>> glance I think so.
> >> >>>
> >> >> Not sure I can follow completely but that hint with*
> >> >> BuiltInsWithParseTimeParameters* got me started, but at the moment
> I'm
> >> >> stuck as I need to get more familiar with the internals of
> Freemarker.
> >> I am
> >> >> also not sure I am on the same page regarding the syntax we are
> aiming
> >> for
> >> >> and why I would need to extend the parser when there is something
> like
> >> >> BuiltInsWithParseTimeParameters....
> >> >
> >> > I assumed that the syntax will be similar to the Java lambda syntax
> >> > (see below why), and that's of course needs lexer/parser changes.
> >> >
> >> >> Here is an example what I have in mind:
> >> >> I started with the ?filter() builtin. I had a syntax like this in
> mind:
> >> >>
> >> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
> >> >> *Example 2: ["a","b","c"]?filter(element, element ==
> someOtherVariable,
> >> >> someOtherVariable="c")*
> >> >
> >> > Looking at the above one believe that the value of `element` is passed
> >> > in as first argument, and the the value of `element == "c"` as the
> >> > second, but that's not the case. It's much better if it's visible that
> >> > you got some kind of anonymous function definition there without
> >> > knowing about the "filter" built-in.
> >> >
> >> > Java already has a syntax for expressing this kind of thing, so it
> >> > would be better to use that familiar syntax. As far as I know it
> >> > doesn't conflict with ours (it kind of it does, but not fatally).
> >> >
> >> > Also, even if for now we only allow this in said built-ins, we
> >> > shouldn't exclude the possibility of making this kind of expression
> >> > accessible elsewhere as well. Then, on many places the parsed won't
> >> > know what kind of value is expected (consider passing a lambda to an
> >> > user defined directive for example), so a syntax like above won't
> >> > work.
> >> >
> >> >> Not sure if that's what you have in mind too, but to me it made sense
> >> with
> >> >> regards to BuiltInsWithParseTimeParameters and I could start without
> >> >> touching parser stuff.
> >> >
> >> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
> >> > bit...), as a call to a such built-in looks like a call to any other
> >> > built-in.
> >> >
> >> >> *1st argument 'element'* would just be the iterator variable similar
> to
> >> >> <#list ["a","b","c"] as *element*>
> >> >> 2nd argument is the filter lambda expression... aka our filter
> condition
> >> >> 3rd+n argument are optional parameters in case used in the lambda
> >> expression
> >> >
> >> > #list is special, similarly as `for ... in ...` is in most languages.
> >> > It's not lambda-ish either; you can't write `as element/2` or such.
> >> >
> >> >> So at first I was looking how <#list> works and found IteratorBlock,
> and
> >> >> though I could reuse it somehow.
> >> >>
> >> >> Here is some simple pseudo code I played around for the for Example
> 1:
> >> >>
> >> >> static class filter_BI extends BuiltInWithParseTimeParameters {
> >> >>
> >> >>
> >> >>
> >> >>         TemplateModel _eval(Environment env) throws
> TemplateException {
> >> >>
> >> >>             // sequence
> >> >>
> >> >>         TemplateModel targetValue = target.evalToNonMissing(env);
> >> >>
> >> >>
> >> >>
> >> >>             List parameters = this.parameters;
> >> >>
> >> >>
> >> >>
> >> >>             Expression iteratorAlias = (Expression)
> parameters.get(0);
> >> >>
> >> >>             Expression conditionExpression = (Expression)
> >> parameters.get(1);
> >> >>
> >> >>
> >> >>
> >> >>             TemplateSequenceModel seq =  (TemplateSequenceModel)
> >> target.eval
> >> >> (env);
> >> >>
> >> >>             for (int i = 0; i < seq.size(); i++) {
> >> >>
> >> >>                TemplateModel cur = seq.get(i);
> >> >>
> >> >>
> >> >>                // this is where I am stuck at the moment
> >> >>
> >> >>                // I basically want to evaluate conditionExpression
> >> >> where iteratorAlias
> >> >> is basically what I passed as 'element'
> >> >>
> >> >>                // I am not sure if or how LocalContext could come
> into
> >> play
> >> >> here
> >> >>
> >> >>                // basically for each iteration I would assign the
> >> current
> >> >> loop element to a context variable with the name 'element'
> >> >>
> >> >>                // and then evaluate conditionExpression with that
> >> context.
> >> >>
> >> >>                // if conditionExpression is "true" then I would
> populate
> >> >> add the current sequence element 'cur'
> >> >>
> >> >>                // to a new result-List.... and return that....
> something
> >> >>
> >> >>                // I wanted to reuse IteratorBlock here somehow, but
> >> didn't
> >> >> get it to work yet.
> >> >>
> >> >>                // maybe this is a stupid idea, or we just need
> something
> >> >> similar
> >> >>
> >> >>
> >> >>
> >> >>             }
> >> >>
> >> >> }
> >> >>
> >> >>
> >> >>
> >> >> Ok so far for my pseudo code....  Maybe you could give some more
> >> pointers
> >> >> based on that... in case this makes any sense ...
> >> >
> >> > For LocalContext, Environment has a pushLocalContext method. See the
> >> > calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
> >> > Map, List).
> >> >
> >> > To avoid running into more new things at once than necessary, perhaps
> >> > you should start with seq?seq_contains(lambda). The end result should
> >> > be like:
> >> >
> >> >   users?contains(u -> u.paying)
> >> >
> >> >>> Another similarity to BuiltInsWithParseTimeParameters is that we
> won't
> >> >>> allow separating the `?someBI` and `(arg)`. Like, with the example
> of
> >> >>> `cond?then(1, 2)`, you aren't allowed to do this:
> >> >>>
> >> >>>   <#assign t=cond?then>
> >> >>>   ${t(1, 2)}
> >> >>>
> >> >>> That maybe looks natural, but most other built-ins allow that. Of
> >> >>> course we can't allow that for ?filter etc., because then we need
> >> >>> closures again.
> >> >>>
> >> >>> >> Multi-part macro body is also planned. Means, I know it
> definitely
> >> >>> >> should be added, but who knows when that's done... I mean, it's
> like
> >> >>> >> that for what, a decade? (: It's not even decided what it exactly
> >> >>> >> does, as there are many ways of approaching this. (I have my own
> >> idea
> >> >>> >> about what the right compromise would be, but others has other
> >> >>> >> ideas...)
> >> >>> >>
> >> >>> >> Filtering lists bothers me because the template language should
> be
> >> >>> >> (and somewhat indeed is) specialized on listing things on fancy
> ways
> >> >>> >> that used to come up when generating document-like output. (If it
> >> >>> >> doesn't do things like that, you might as well use a general
> purpose
> >> >>> >> language.) Thus, that filter is unsolved (filtering with #if is
> >> >>> >> verbose and spoils #sep etc.) bothers me a lot.
> >> >>> >>
> >> >>> >> BTW, ?filter and ?map is also especially handy in our case as
> >> >>> >> FreeMarker doesn't support building new sequences (sequences are
> >> >>> >> immutable). Although it has sequence concatenation with `+`, it's
> >> not
> >> >>> >> good for building a sequence one by one, unless the sequence
> will be
> >> >>> >> quite short.
> >> >>> >>
> >> >>> > Good point.
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >> > Cheers,
> >> >>> >> > -- Denis.
> >> >>> >>
> >> >>> >> --
> >> >>> >> Thanks,
> >> >>> >>  Daniel Dekany
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>>
> >> >>> --
> >> >>> Thanks,
> >> >>>  Daniel Dekany
> >> >>>
> >> >>>
> >> >> Thanks
> >> >> Christoph
> >> >>
> >> >
> >>
> >> --
> >> Thanks,
> >>  Daniel Dekany
> >>
> >>
> >
> > --
> > Christoph Rüger, Geschäftsführer
> > Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
> > Programmieren - Automatisierung, Schnittstellen, Datenfeeds
> >
> > Xing: https://www.xing.com/profile/Christoph_Rueger2
> > LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
> >
>
> --
> Thanks,
>  Daniel Dekany
>
>

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 
5596493Internet: https://synesty.com <https://synesty.com>
Informationen 
zum Datenschutz: https://synesty.com/datenschutz 
<https://synesty.com/datenschutz>

Geschäftsführer: Christoph Rüger

Unternehmenssitz: Jena
Handelsregister B beim Amtsgericht: Jena

Handelsregister-Nummer: HRB 508766
Ust-IdNr.: DE287564982

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

I have pushed an implementation of the restricted lambdas we were
talking about. Guys, please review/test it.

I call these "local lambdas" in the source code (but I'm open for
suggestions), as the function they define can only be called in the
same variable scope where the lambda was (as we have no closures in
the template language, nor final variables). Also, they can only be
used as the parameters of the built-ins that explicitly support them.
As the subject shows, the main goal was just support filtering for
#list, without a nested #if, as that breaks #sep, #else, it?hasNext,
it?index, etc. (I will try to make more universal lambdas in FM3... I
guess it would be way too tricky in FM2.)

For now, I have only added two built-ins that support lambdas: ?filter
and ?map. Any ideas what other such built-ins would be often useful in
templates (with use case, if possible)?

Examples of using ?filter and ?map:

  <#list products?filter(it -> it.price < 1000) as product>
    ${product.name}
  </#list>

  <#list products?map(it -> it.name) as name>
    ${name}
  </#list>

Of course these built-ins aren't specific to #list, they can be used
anywhere. Naturally, they can be chained as well:

  <#assign chepProdNames = products
     ?filter(it -> it.price < 1000)
     ?map(it -> it.name)
  >

As a side note, ?filter and ?map also accepts FTL function-s and Java
methods as its parameter, not only lambdas.

A tricky aspect of this feature is lazy evaluation, in similar sense
as Java 8 Stream intermediate operations are lazy. On most places,
?filer and ?map are eager, that is, they return a completed sequence
of items. That's because our restricted lambdas only work correctly
"locally". However, there are very common situations where it's clear
that we can use lazy ("streaming") evaluation, as we know where the
resulting stream of elements will be consumed:

- One such case is when these kind of built-ins are chained. Like in
  the last example, ?filter doesn't construct a List in memory,
  instead the elements just flow through it (some is dropped, as it's
  filtering), into ?map. Only the ?map at the end of the chain will
  built a List eagerly. Some other built-ins also allow the left-hand
  built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
  is built anywhere, instead the elements flow through both ?filter
  and ?map, and ?join just appends them to the StringBuilder where it
  creates its results.

- #list also enables lazy evaluation to its 1st parameter. So in the
  #list examples above, yet again no List is built in memory.

- There are other such cases, which I didn't implement yet. For
  example the sequence slice operator, like, xs?filter(f)[10..20],
  should allow lazy evaluation.

Feedback is welcome!


Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:

> Hey Daniel,
> I'm very sorry, but I didn't make any progress with this. I think I was a
> bit over-motivated, but unfortunately I cannot spend more time on this for
> various reasons.
> You can take this over. I'm glad to help out with testing and feedback.
>
> Thanks
> Christoph
>
>
>
>
> Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>:
>
>> Any progress in this? I think I will give it a try in the coming days
>> otherwise.
>>
>>
>> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
>>
>> > See my answers inline...
>> >
>> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
>> >
>> >> Thanks Daniel for your feedback. See my answers below
>> >>
>> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
>> ddekany@apache.org
>> >>>:
>> >>
>> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>> >>>
>> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
>> >>> ddekany@apache.org
>> >>> >>:
>> >>> >
>> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>> >>> >>
>> >>> >> > Hi,
>> >>> >> >
>> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com>
>> a
>> >>> >> écrit :
>> >>> >> >
>> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>> >>> >> ddekany@apache.org
>> >>> >> > :
>> >>> >> >
>> >>> >> > It's certainly tricky, but as far as I see possible (but then, who
>> >>> >> >
>> >>> >> > knows what will one find when actually working on it). It's also a
>> >>> >> > feature missing a lot. It's especially missing for #list (I know
>> that
>> >>> >> > you need it for something else), because if you filter the items
>> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be
>> usable.
>> >>> >> >
>> >>> >> > Let me say that I disagree here.
>> >>> >> >
>> >>> >> > I do not think that closures are required for FreeMarker, nor that
>> >>> they
>> >>> >> are a good idea.
>> >>> >> >
>> >>> >> > If we add new features to the FreeMarker *tempate engine* I would
>> >>> >> > rather we focus on multi-part macro body rather than an advanced
>> >>> >> language feature like closures.
>> >>> >> >
>> >>> >> > You can add ?filter and ?map if you want, a simple expression as
>> >>> >> parameter should be enough.
>> >>> >>
>> >>> >> Yes, as I said, we certainly start with only allowing lambdas in
>> >>> >> ?filter/?map, also certainly in ?contains.
>> >>> >>
>> >>> > Would be enough in my opinion and very useful.
>> >>> >
>> >>> > Is it possiblefor you to give some pointers to the code on how this
>> could
>> >>> > be implemented? I would maybe like to wrap my head around this a
>> little
>> >>> bit.
>> >>>
>> >>> Please feel yourself encouraged! (:
>> >>>
>> >>> > I started looking at seq_containsBI (
>> >>> >
>> >>>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
>> >>> )
>> >>> > and
>> >>> > and reverseBI (
>> >>> >
>> >>>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
>> >>> )
>> >>> > just to find something related (seq_containsBI checks something) and
>> >>> > reverseBI returns a new sequence.
>> >>> > What I haven't found is a function which takes an Expression as a
>> >>> > parameter.
>> >>> > Is there something similar already or would that be a new thing?
>> >>>
>> >>> It's a new thing in that it will be part of the expression syntax
>> >>> (even if for now we will only allow lambdas as the parameters of a few
>> >>> built-ins, so that we can get away without closures). So it's a new
>> >>> Expression subclass, and has to be part of the parser (ftl.jj) as
>> >>> well.
>> >>
>> >> Hmm, that parser stuff is new for me, it'll take me some time to get
>> into
>> >> it.
>> >
>> > So it's a JavaCC lexer+parser. With a few twists... sorry, but this
>> > code has history... :)
>> >
>> >>> As of lazy evaluation of parameters expressions, that's already
>> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and you will
>> >>> see it's trivial to do, but the situation there is much simpler.
>> >>>
>> >>
>> >>
>> >>>
>> >>> In principle, a LambdaExpression should evaluate to a
>> >>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
>> >>> the called built-in or whatever it is. But with the approach of
>> >>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
>> >>> just bind to the LambdaExpression directly, add a LocalContext that
>> >>> contains the lambda arguments, end evaluate the LambdaExpression right
>> >>> there, in the built-in implementation. Or at least at a very quick
>> >>> glance I think so.
>> >>>
>> >> Not sure I can follow completely but that hint with*
>> >> BuiltInsWithParseTimeParameters* got me started, but at the moment I'm
>> >> stuck as I need to get more familiar with the internals of Freemarker.
>> I am
>> >> also not sure I am on the same page regarding the syntax we are aiming
>> for
>> >> and why I would need to extend the parser when there is something like
>> >> BuiltInsWithParseTimeParameters....
>> >
>> > I assumed that the syntax will be similar to the Java lambda syntax
>> > (see below why), and that's of course needs lexer/parser changes.
>> >
>> >> Here is an example what I have in mind:
>> >> I started with the ?filter() builtin. I had a syntax like this in mind:
>> >>
>> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
>> >> *Example 2: ["a","b","c"]?filter(element, element == someOtherVariable,
>> >> someOtherVariable="c")*
>> >
>> > Looking at the above one believe that the value of `element` is passed
>> > in as first argument, and the the value of `element == "c"` as the
>> > second, but that's not the case. It's much better if it's visible that
>> > you got some kind of anonymous function definition there without
>> > knowing about the "filter" built-in.
>> >
>> > Java already has a syntax for expressing this kind of thing, so it
>> > would be better to use that familiar syntax. As far as I know it
>> > doesn't conflict with ours (it kind of it does, but not fatally).
>> >
>> > Also, even if for now we only allow this in said built-ins, we
>> > shouldn't exclude the possibility of making this kind of expression
>> > accessible elsewhere as well. Then, on many places the parsed won't
>> > know what kind of value is expected (consider passing a lambda to an
>> > user defined directive for example), so a syntax like above won't
>> > work.
>> >
>> >> Not sure if that's what you have in mind too, but to me it made sense
>> with
>> >> regards to BuiltInsWithParseTimeParameters and I could start without
>> >> touching parser stuff.
>> >
>> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
>> > bit...), as a call to a such built-in looks like a call to any other
>> > built-in.
>> >
>> >> *1st argument 'element'* would just be the iterator variable similar to
>> >> <#list ["a","b","c"] as *element*>
>> >> 2nd argument is the filter lambda expression... aka our filter condition
>> >> 3rd+n argument are optional parameters in case used in the lambda
>> expression
>> >
>> > #list is special, similarly as `for ... in ...` is in most languages.
>> > It's not lambda-ish either; you can't write `as element/2` or such.
>> >
>> >> So at first I was looking how <#list> works and found IteratorBlock, and
>> >> though I could reuse it somehow.
>> >>
>> >> Here is some simple pseudo code I played around for the for Example 1:
>> >>
>> >> static class filter_BI extends BuiltInWithParseTimeParameters {
>> >>
>> >>
>> >>
>> >>         TemplateModel _eval(Environment env) throws TemplateException {
>> >>
>> >>             // sequence
>> >>
>> >>         TemplateModel targetValue = target.evalToNonMissing(env);
>> >>
>> >>
>> >>
>> >>             List parameters = this.parameters;
>> >>
>> >>
>> >>
>> >>             Expression iteratorAlias = (Expression) parameters.get(0);
>> >>
>> >>             Expression conditionExpression = (Expression)
>> parameters.get(1);
>> >>
>> >>
>> >>
>> >>             TemplateSequenceModel seq =  (TemplateSequenceModel)
>> target.eval
>> >> (env);
>> >>
>> >>             for (int i = 0; i < seq.size(); i++) {
>> >>
>> >>                TemplateModel cur = seq.get(i);
>> >>
>> >>
>> >>                // this is where I am stuck at the moment
>> >>
>> >>                // I basically want to evaluate conditionExpression
>> >> where iteratorAlias
>> >> is basically what I passed as 'element'
>> >>
>> >>                // I am not sure if or how LocalContext could come into
>> play
>> >> here
>> >>
>> >>                // basically for each iteration I would assign the
>> current
>> >> loop element to a context variable with the name 'element'
>> >>
>> >>                // and then evaluate conditionExpression with that
>> context.
>> >>
>> >>                // if conditionExpression is "true" then I would populate
>> >> add the current sequence element 'cur'
>> >>
>> >>                // to a new result-List.... and return that.... something
>> >>
>> >>                // I wanted to reuse IteratorBlock here somehow, but
>> didn't
>> >> get it to work yet.
>> >>
>> >>                // maybe this is a stupid idea, or we just need something
>> >> similar
>> >>
>> >>
>> >>
>> >>             }
>> >>
>> >> }
>> >>
>> >>
>> >>
>> >> Ok so far for my pseudo code....  Maybe you could give some more
>> pointers
>> >> based on that... in case this makes any sense ...
>> >
>> > For LocalContext, Environment has a pushLocalContext method. See the
>> > calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
>> > Map, List).
>> >
>> > To avoid running into more new things at once than necessary, perhaps
>> > you should start with seq?seq_contains(lambda). The end result should
>> > be like:
>> >
>> >   users?contains(u -> u.paying)
>> >
>> >>> Another similarity to BuiltInsWithParseTimeParameters is that we won't
>> >>> allow separating the `?someBI` and `(arg)`. Like, with the example of
>> >>> `cond?then(1, 2)`, you aren't allowed to do this:
>> >>>
>> >>>   <#assign t=cond?then>
>> >>>   ${t(1, 2)}
>> >>>
>> >>> That maybe looks natural, but most other built-ins allow that. Of
>> >>> course we can't allow that for ?filter etc., because then we need
>> >>> closures again.
>> >>>
>> >>> >> Multi-part macro body is also planned. Means, I know it definitely
>> >>> >> should be added, but who knows when that's done... I mean, it's like
>> >>> >> that for what, a decade? (: It's not even decided what it exactly
>> >>> >> does, as there are many ways of approaching this. (I have my own
>> idea
>> >>> >> about what the right compromise would be, but others has other
>> >>> >> ideas...)
>> >>> >>
>> >>> >> Filtering lists bothers me because the template language should be
>> >>> >> (and somewhat indeed is) specialized on listing things on fancy ways
>> >>> >> that used to come up when generating document-like output. (If it
>> >>> >> doesn't do things like that, you might as well use a general purpose
>> >>> >> language.) Thus, that filter is unsolved (filtering with #if is
>> >>> >> verbose and spoils #sep etc.) bothers me a lot.
>> >>> >>
>> >>> >> BTW, ?filter and ?map is also especially handy in our case as
>> >>> >> FreeMarker doesn't support building new sequences (sequences are
>> >>> >> immutable). Although it has sequence concatenation with `+`, it's
>> not
>> >>> >> good for building a sequence one by one, unless the sequence will be
>> >>> >> quite short.
>> >>> >>
>> >>> > Good point.
>> >>> >
>> >>> >
>> >>> >>
>> >>> >> > Cheers,
>> >>> >> > -- Denis.
>> >>> >>
>> >>> >> --
>> >>> >> Thanks,
>> >>> >>  Daniel Dekany
>> >>> >>
>> >>> >>
>> >>> >
>> >>>
>> >>> --
>> >>> Thanks,
>> >>>  Daniel Dekany
>> >>>
>> >>>
>> >> Thanks
>> >> Christoph
>> >>
>> >
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
>
> -- 
> Christoph Rüger, Geschäftsführer
> Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
> Programmieren - Automatisierung, Schnittstellen, Datenfeeds
>
> Xing: https://www.xing.com/profile/Christoph_Rueger2
> LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
>

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

Hey Daniel,
I'm very sorry, but I didn't make any progress with this. I think I was a
bit over-motivated, but unfortunately I cannot spend more time on this for
various reasons.
You can take this over. I'm glad to help out with testing and feedback.

Thanks
Christoph




Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <ddekany@apache.org
>:

> Any progress in this? I think I will give it a try in the coming days
> otherwise.
>
>
> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
>
> > See my answers inline...
> >
> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
> >
> >> Thanks Daniel for your feedback. See my answers below
> >>
> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
> ddekany@apache.org
> >>>:
> >>
> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
> >>>
> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
> >>> ddekany@apache.org
> >>> >>:
> >>> >
> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
> >>> >>
> >>> >> > Hi,
> >>> >> >
> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com>
> a
> >>> >> écrit :
> >>> >> >
> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
> >>> >> ddekany@apache.org
> >>> >> > :
> >>> >> >
> >>> >> > It's certainly tricky, but as far as I see possible (but then, who
> >>> >> >
> >>> >> > knows what will one find when actually working on it). It's also a
> >>> >> > feature missing a lot. It's especially missing for #list (I know
> that
> >>> >> > you need it for something else), because if you filter the items
> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be
> usable.
> >>> >> >
> >>> >> > Let me say that I disagree here.
> >>> >> >
> >>> >> > I do not think that closures are required for FreeMarker, nor that
> >>> they
> >>> >> are a good idea.
> >>> >> >
> >>> >> > If we add new features to the FreeMarker *tempate engine* I would
> >>> >> > rather we focus on multi-part macro body rather than an advanced
> >>> >> language feature like closures.
> >>> >> >
> >>> >> > You can add ?filter and ?map if you want, a simple expression as
> >>> >> parameter should be enough.
> >>> >>
> >>> >> Yes, as I said, we certainly start with only allowing lambdas in
> >>> >> ?filter/?map, also certainly in ?contains.
> >>> >>
> >>> > Would be enough in my opinion and very useful.
> >>> >
> >>> > Is it possiblefor you to give some pointers to the code on how this
> could
> >>> > be implemented? I would maybe like to wrap my head around this a
> little
> >>> bit.
> >>>
> >>> Please feel yourself encouraged! (:
> >>>
> >>> > I started looking at seq_containsBI (
> >>> >
> >>>
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
> >>> )
> >>> > and
> >>> > and reverseBI (
> >>> >
> >>>
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
> >>> )
> >>> > just to find something related (seq_containsBI checks something) and
> >>> > reverseBI returns a new sequence.
> >>> > What I haven't found is a function which takes an Expression as a
> >>> > parameter.
> >>> > Is there something similar already or would that be a new thing?
> >>>
> >>> It's a new thing in that it will be part of the expression syntax
> >>> (even if for now we will only allow lambdas as the parameters of a few
> >>> built-ins, so that we can get away without closures). So it's a new
> >>> Expression subclass, and has to be part of the parser (ftl.jj) as
> >>> well.
> >>
> >> Hmm, that parser stuff is new for me, it'll take me some time to get
> into
> >> it.
> >
> > So it's a JavaCC lexer+parser. With a few twists... sorry, but this
> > code has history... :)
> >
> >>> As of lazy evaluation of parameters expressions, that's already
> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and you will
> >>> see it's trivial to do, but the situation there is much simpler.
> >>>
> >>
> >>
> >>>
> >>> In principle, a LambdaExpression should evaluate to a
> >>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
> >>> the called built-in or whatever it is. But with the approach of
> >>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
> >>> just bind to the LambdaExpression directly, add a LocalContext that
> >>> contains the lambda arguments, end evaluate the LambdaExpression right
> >>> there, in the built-in implementation. Or at least at a very quick
> >>> glance I think so.
> >>>
> >> Not sure I can follow completely but that hint with*
> >> BuiltInsWithParseTimeParameters* got me started, but at the moment I'm
> >> stuck as I need to get more familiar with the internals of Freemarker.
> I am
> >> also not sure I am on the same page regarding the syntax we are aiming
> for
> >> and why I would need to extend the parser when there is something like
> >> BuiltInsWithParseTimeParameters....
> >
> > I assumed that the syntax will be similar to the Java lambda syntax
> > (see below why), and that's of course needs lexer/parser changes.
> >
> >> Here is an example what I have in mind:
> >> I started with the ?filter() builtin. I had a syntax like this in mind:
> >>
> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
> >> *Example 2: ["a","b","c"]?filter(element, element == someOtherVariable,
> >> someOtherVariable="c")*
> >
> > Looking at the above one believe that the value of `element` is passed
> > in as first argument, and the the value of `element == "c"` as the
> > second, but that's not the case. It's much better if it's visible that
> > you got some kind of anonymous function definition there without
> > knowing about the "filter" built-in.
> >
> > Java already has a syntax for expressing this kind of thing, so it
> > would be better to use that familiar syntax. As far as I know it
> > doesn't conflict with ours (it kind of it does, but not fatally).
> >
> > Also, even if for now we only allow this in said built-ins, we
> > shouldn't exclude the possibility of making this kind of expression
> > accessible elsewhere as well. Then, on many places the parsed won't
> > know what kind of value is expected (consider passing a lambda to an
> > user defined directive for example), so a syntax like above won't
> > work.
> >
> >> Not sure if that's what you have in mind too, but to me it made sense
> with
> >> regards to BuiltInsWithParseTimeParameters and I could start without
> >> touching parser stuff.
> >
> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
> > bit...), as a call to a such built-in looks like a call to any other
> > built-in.
> >
> >> *1st argument 'element'* would just be the iterator variable similar to
> >> <#list ["a","b","c"] as *element*>
> >> 2nd argument is the filter lambda expression... aka our filter condition
> >> 3rd+n argument are optional parameters in case used in the lambda
> expression
> >
> > #list is special, similarly as `for ... in ...` is in most languages.
> > It's not lambda-ish either; you can't write `as element/2` or such.
> >
> >> So at first I was looking how <#list> works and found IteratorBlock, and
> >> though I could reuse it somehow.
> >>
> >> Here is some simple pseudo code I played around for the for Example 1:
> >>
> >> static class filter_BI extends BuiltInWithParseTimeParameters {
> >>
> >>
> >>
> >>         TemplateModel _eval(Environment env) throws TemplateException {
> >>
> >>             // sequence
> >>
> >>         TemplateModel targetValue = target.evalToNonMissing(env);
> >>
> >>
> >>
> >>             List parameters = this.parameters;
> >>
> >>
> >>
> >>             Expression iteratorAlias = (Expression) parameters.get(0);
> >>
> >>             Expression conditionExpression = (Expression)
> parameters.get(1);
> >>
> >>
> >>
> >>             TemplateSequenceModel seq =  (TemplateSequenceModel)
> target.eval
> >> (env);
> >>
> >>             for (int i = 0; i < seq.size(); i++) {
> >>
> >>                TemplateModel cur = seq.get(i);
> >>
> >>
> >>                // this is where I am stuck at the moment
> >>
> >>                // I basically want to evaluate conditionExpression
> >> where iteratorAlias
> >> is basically what I passed as 'element'
> >>
> >>                // I am not sure if or how LocalContext could come into
> play
> >> here
> >>
> >>                // basically for each iteration I would assign the
> current
> >> loop element to a context variable with the name 'element'
> >>
> >>                // and then evaluate conditionExpression with that
> context.
> >>
> >>                // if conditionExpression is "true" then I would populate
> >> add the current sequence element 'cur'
> >>
> >>                // to a new result-List.... and return that.... something
> >>
> >>                // I wanted to reuse IteratorBlock here somehow, but
> didn't
> >> get it to work yet.
> >>
> >>                // maybe this is a stupid idea, or we just need something
> >> similar
> >>
> >>
> >>
> >>             }
> >>
> >> }
> >>
> >>
> >>
> >> Ok so far for my pseudo code....  Maybe you could give some more
> pointers
> >> based on that... in case this makes any sense ...
> >
> > For LocalContext, Environment has a pushLocalContext method. See the
> > calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
> > Map, List).
> >
> > To avoid running into more new things at once than necessary, perhaps
> > you should start with seq?seq_contains(lambda). The end result should
> > be like:
> >
> >   users?contains(u -> u.paying)
> >
> >>> Another similarity to BuiltInsWithParseTimeParameters is that we won't
> >>> allow separating the `?someBI` and `(arg)`. Like, with the example of
> >>> `cond?then(1, 2)`, you aren't allowed to do this:
> >>>
> >>>   <#assign t=cond?then>
> >>>   ${t(1, 2)}
> >>>
> >>> That maybe looks natural, but most other built-ins allow that. Of
> >>> course we can't allow that for ?filter etc., because then we need
> >>> closures again.
> >>>
> >>> >> Multi-part macro body is also planned. Means, I know it definitely
> >>> >> should be added, but who knows when that's done... I mean, it's like
> >>> >> that for what, a decade? (: It's not even decided what it exactly
> >>> >> does, as there are many ways of approaching this. (I have my own
> idea
> >>> >> about what the right compromise would be, but others has other
> >>> >> ideas...)
> >>> >>
> >>> >> Filtering lists bothers me because the template language should be
> >>> >> (and somewhat indeed is) specialized on listing things on fancy ways
> >>> >> that used to come up when generating document-like output. (If it
> >>> >> doesn't do things like that, you might as well use a general purpose
> >>> >> language.) Thus, that filter is unsolved (filtering with #if is
> >>> >> verbose and spoils #sep etc.) bothers me a lot.
> >>> >>
> >>> >> BTW, ?filter and ?map is also especially handy in our case as
> >>> >> FreeMarker doesn't support building new sequences (sequences are
> >>> >> immutable). Although it has sequence concatenation with `+`, it's
> not
> >>> >> good for building a sequence one by one, unless the sequence will be
> >>> >> quite short.
> >>> >>
> >>> > Good point.
> >>> >
> >>> >
> >>> >>
> >>> >> > Cheers,
> >>> >> > -- Denis.
> >>> >>
> >>> >> --
> >>> >> Thanks,
> >>> >>  Daniel Dekany
> >>> >>
> >>> >>
> >>> >
> >>>
> >>> --
> >>> Thanks,
> >>>  Daniel Dekany
> >>>
> >>>
> >> Thanks
> >> Christoph
> >>
> >
>
> --
> Thanks,
>  Daniel Dekany
>
>

-- 
Christoph Rüger, Geschäftsführer
Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
Programmieren - Automatisierung, Schnittstellen, Datenfeeds

Xing: https://www.xing.com/profile/Christoph_Rueger2
LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 5596493Fax.: 
+49 3641 5596499
Internet: https://synesty.com <https://synesty.com>


Geschäftsführer: Christoph Rüger
Unternehmenssitz: Jena
Handelsregister B 
beim Amtsgericht: Jena
Handelsregister-Nummer: HRB 508766
Ust-IdNr.: 
DE287564982

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

Any progress in this? I think I will give it a try in the coming days
otherwise.


Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:

> See my answers inline...
>
> Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
>
>> Thanks Daniel for your feedback. See my answers below
>>
>> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>>:
>>
>>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>>>
>>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
>>> ddekany@apache.org
>>> >>:
>>> >
>>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>>> >>
>>> >> > Hi,
>>> >> >
>>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com> a
>>> >> écrit :
>>> >> >
>>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>>> >> ddekany@apache.org
>>> >> > :
>>> >> >
>>> >> > It's certainly tricky, but as far as I see possible (but then, who
>>> >> >
>>> >> > knows what will one find when actually working on it). It's also a
>>> >> > feature missing a lot. It's especially missing for #list (I know that
>>> >> > you need it for something else), because if you filter the items
>>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be usable.
>>> >> >
>>> >> > Let me say that I disagree here.
>>> >> >
>>> >> > I do not think that closures are required for FreeMarker, nor that
>>> they
>>> >> are a good idea.
>>> >> >
>>> >> > If we add new features to the FreeMarker *tempate engine* I would
>>> >> > rather we focus on multi-part macro body rather than an advanced
>>> >> language feature like closures.
>>> >> >
>>> >> > You can add ?filter and ?map if you want, a simple expression as
>>> >> parameter should be enough.
>>> >>
>>> >> Yes, as I said, we certainly start with only allowing lambdas in
>>> >> ?filter/?map, also certainly in ?contains.
>>> >>
>>> > Would be enough in my opinion and very useful.
>>> >
>>> > Is it possiblefor you to give some pointers to the code on how this could
>>> > be implemented? I would maybe like to wrap my head around this a little
>>> bit.
>>>
>>> Please feel yourself encouraged! (:
>>>
>>> > I started looking at seq_containsBI (
>>> >
>>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
>>> )
>>> > and
>>> > and reverseBI (
>>> >
>>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
>>> )
>>> > just to find something related (seq_containsBI checks something) and
>>> > reverseBI returns a new sequence.
>>> > What I haven't found is a function which takes an Expression as a
>>> > parameter.
>>> > Is there something similar already or would that be a new thing?
>>>
>>> It's a new thing in that it will be part of the expression syntax
>>> (even if for now we will only allow lambdas as the parameters of a few
>>> built-ins, so that we can get away without closures). So it's a new
>>> Expression subclass, and has to be part of the parser (ftl.jj) as
>>> well.
>>
>> Hmm, that parser stuff is new for me, it'll take me some time to get into
>> it.
>
> So it's a JavaCC lexer+parser. With a few twists... sorry, but this
> code has history... :)
>
>>> As of lazy evaluation of parameters expressions, that's already
>>> done in the built-ins in BuiltInsWithParseTimeParameters, and you will
>>> see it's trivial to do, but the situation there is much simpler.
>>>
>>
>>
>>>
>>> In principle, a LambdaExpression should evaluate to a
>>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
>>> the called built-in or whatever it is. But with the approach of
>>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
>>> just bind to the LambdaExpression directly, add a LocalContext that
>>> contains the lambda arguments, end evaluate the LambdaExpression right
>>> there, in the built-in implementation. Or at least at a very quick
>>> glance I think so.
>>>
>> Not sure I can follow completely but that hint with*
>> BuiltInsWithParseTimeParameters* got me started, but at the moment I'm
>> stuck as I need to get more familiar with the internals of Freemarker. I am
>> also not sure I am on the same page regarding the syntax we are aiming for
>> and why I would need to extend the parser when there is something like
>> BuiltInsWithParseTimeParameters....
>
> I assumed that the syntax will be similar to the Java lambda syntax
> (see below why), and that's of course needs lexer/parser changes.
>
>> Here is an example what I have in mind:
>> I started with the ?filter() builtin. I had a syntax like this in mind:
>>
>> *Example 1: ["a","b","c"]?filter(element, element == "c")*
>> *Example 2: ["a","b","c"]?filter(element, element == someOtherVariable,
>> someOtherVariable="c")*
>
> Looking at the above one believe that the value of `element` is passed
> in as first argument, and the the value of `element == "c"` as the
> second, but that's not the case. It's much better if it's visible that
> you got some kind of anonymous function definition there without
> knowing about the "filter" built-in.
>
> Java already has a syntax for expressing this kind of thing, so it
> would be better to use that familiar syntax. As far as I know it
> doesn't conflict with ours (it kind of it does, but not fatally).
>
> Also, even if for now we only allow this in said built-ins, we
> shouldn't exclude the possibility of making this kind of expression
> accessible elsewhere as well. Then, on many places the parsed won't
> know what kind of value is expected (consider passing a lambda to an
> user defined directive for example), so a syntax like above won't
> work.
>
>> Not sure if that's what you have in mind too, but to me it made sense with
>> regards to BuiltInsWithParseTimeParameters and I could start without
>> touching parser stuff.
>
> BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
> bit...), as a call to a such built-in looks like a call to any other
> built-in.
>
>> *1st argument 'element'* would just be the iterator variable similar to
>> <#list ["a","b","c"] as *element*>
>> 2nd argument is the filter lambda expression... aka our filter condition
>> 3rd+n argument are optional parameters in case used in the lambda expression
>
> #list is special, similarly as `for ... in ...` is in most languages.
> It's not lambda-ish either; you can't write `as element/2` or such.
>
>> So at first I was looking how <#list> works and found IteratorBlock, and
>> though I could reuse it somehow.
>>
>> Here is some simple pseudo code I played around for the for Example 1:
>>
>> static class filter_BI extends BuiltInWithParseTimeParameters {
>>
>>
>>
>>         TemplateModel _eval(Environment env) throws TemplateException {
>>
>>             // sequence
>>
>>         TemplateModel targetValue = target.evalToNonMissing(env);
>>
>>
>>
>>             List parameters = this.parameters;
>>
>>
>>
>>             Expression iteratorAlias = (Expression) parameters.get(0);
>>
>>             Expression conditionExpression = (Expression) parameters.get(1);
>>
>>
>>
>>             TemplateSequenceModel seq =  (TemplateSequenceModel) target.eval
>> (env);
>>
>>             for (int i = 0; i < seq.size(); i++) {
>>
>>                TemplateModel cur = seq.get(i);
>>
>>
>>                // this is where I am stuck at the moment
>>
>>                // I basically want to evaluate conditionExpression
>> where iteratorAlias
>> is basically what I passed as 'element'
>>
>>                // I am not sure if or how LocalContext could come into play
>> here
>>
>>                // basically for each iteration I would assign the current
>> loop element to a context variable with the name 'element'
>>
>>                // and then evaluate conditionExpression with that context.
>>
>>                // if conditionExpression is "true" then I would populate
>> add the current sequence element 'cur'
>>
>>                // to a new result-List.... and return that.... something
>>
>>                // I wanted to reuse IteratorBlock here somehow, but didn't
>> get it to work yet.
>>
>>                // maybe this is a stupid idea, or we just need something
>> similar
>>
>>
>>
>>             }
>>
>> }
>>
>>
>>
>> Ok so far for my pseudo code....  Maybe you could give some more pointers
>> based on that... in case this makes any sense ...
>
> For LocalContext, Environment has a pushLocalContext method. See the
> calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
> Map, List).
>
> To avoid running into more new things at once than necessary, perhaps
> you should start with seq?seq_contains(lambda). The end result should
> be like:
>
>   users?contains(u -> u.paying)
>
>>> Another similarity to BuiltInsWithParseTimeParameters is that we won't
>>> allow separating the `?someBI` and `(arg)`. Like, with the example of
>>> `cond?then(1, 2)`, you aren't allowed to do this:
>>>
>>>   <#assign t=cond?then>
>>>   ${t(1, 2)}
>>>
>>> That maybe looks natural, but most other built-ins allow that. Of
>>> course we can't allow that for ?filter etc., because then we need
>>> closures again.
>>>
>>> >> Multi-part macro body is also planned. Means, I know it definitely
>>> >> should be added, but who knows when that's done... I mean, it's like
>>> >> that for what, a decade? (: It's not even decided what it exactly
>>> >> does, as there are many ways of approaching this. (I have my own idea
>>> >> about what the right compromise would be, but others has other
>>> >> ideas...)
>>> >>
>>> >> Filtering lists bothers me because the template language should be
>>> >> (and somewhat indeed is) specialized on listing things on fancy ways
>>> >> that used to come up when generating document-like output. (If it
>>> >> doesn't do things like that, you might as well use a general purpose
>>> >> language.) Thus, that filter is unsolved (filtering with #if is
>>> >> verbose and spoils #sep etc.) bothers me a lot.
>>> >>
>>> >> BTW, ?filter and ?map is also especially handy in our case as
>>> >> FreeMarker doesn't support building new sequences (sequences are
>>> >> immutable). Although it has sequence concatenation with `+`, it's not
>>> >> good for building a sequence one by one, unless the sequence will be
>>> >> quite short.
>>> >>
>>> > Good point.
>>> >
>>> >
>>> >>
>>> >> > Cheers,
>>> >> > -- Denis.
>>> >>
>>> >> --
>>> >> Thanks,
>>> >>  Daniel Dekany
>>> >>
>>> >>
>>> >
>>>
>>> --
>>> Thanks,
>>>  Daniel Dekany
>>>
>>>
>> Thanks
>> Christoph
>>
>

-- 
Thanks,
 Daniel Dekany

Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

See my answers inline...

Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:

> Thanks Daniel for your feedback. See my answers below
>
> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>:
>
>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>>
>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
>> ddekany@apache.org
>> >>:
>> >
>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com> a
>> >> écrit :
>> >> >
>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>> >> ddekany@apache.org
>> >> > :
>> >> >
>> >> > It's certainly tricky, but as far as I see possible (but then, who
>> >> >
>> >> > knows what will one find when actually working on it). It's also a
>> >> > feature missing a lot. It's especially missing for #list (I know that
>> >> > you need it for something else), because if you filter the items
>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be usable.
>> >> >
>> >> > Let me say that I disagree here.
>> >> >
>> >> > I do not think that closures are required for FreeMarker, nor that
>> they
>> >> are a good idea.
>> >> >
>> >> > If we add new features to the FreeMarker *tempate engine* I would
>> >> > rather we focus on multi-part macro body rather than an advanced
>> >> language feature like closures.
>> >> >
>> >> > You can add ?filter and ?map if you want, a simple expression as
>> >> parameter should be enough.
>> >>
>> >> Yes, as I said, we certainly start with only allowing lambdas in
>> >> ?filter/?map, also certainly in ?contains.
>> >>
>> > Would be enough in my opinion and very useful.
>> >
>> > Is it possiblefor you to give some pointers to the code on how this could
>> > be implemented? I would maybe like to wrap my head around this a little
>> bit.
>>
>> Please feel yourself encouraged! (:
>>
>> > I started looking at seq_containsBI (
>> >
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
>> )
>> > and
>> > and reverseBI (
>> >
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
>> )
>> > just to find something related (seq_containsBI checks something) and
>> > reverseBI returns a new sequence.
>> > What I haven't found is a function which takes an Expression as a
>> > parameter.
>> > Is there something similar already or would that be a new thing?
>>
>> It's a new thing in that it will be part of the expression syntax
>> (even if for now we will only allow lambdas as the parameters of a few
>> built-ins, so that we can get away without closures). So it's a new
>> Expression subclass, and has to be part of the parser (ftl.jj) as
>> well.
>
> Hmm, that parser stuff is new for me, it'll take me some time to get into
> it.

So it's a JavaCC lexer+parser. With a few twists... sorry, but this
code has history... :)

>> As of lazy evaluation of parameters expressions, that's already
>> done in the built-ins in BuiltInsWithParseTimeParameters, and you will
>> see it's trivial to do, but the situation there is much simpler.
>>
>
>
>>
>> In principle, a LambdaExpression should evaluate to a
>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
>> the called built-in or whatever it is. But with the approach of
>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
>> just bind to the LambdaExpression directly, add a LocalContext that
>> contains the lambda arguments, end evaluate the LambdaExpression right
>> there, in the built-in implementation. Or at least at a very quick
>> glance I think so.
>>
> Not sure I can follow completely but that hint with*
> BuiltInsWithParseTimeParameters* got me started, but at the moment I'm
> stuck as I need to get more familiar with the internals of Freemarker. I am
> also not sure I am on the same page regarding the syntax we are aiming for
> and why I would need to extend the parser when there is something like
> BuiltInsWithParseTimeParameters....

I assumed that the syntax will be similar to the Java lambda syntax
(see below why), and that's of course needs lexer/parser changes.

> Here is an example what I have in mind:
> I started with the ?filter() builtin. I had a syntax like this in mind:
>
> *Example 1: ["a","b","c"]?filter(element, element == "c")*
> *Example 2: ["a","b","c"]?filter(element, element == someOtherVariable,
> someOtherVariable="c")*

Looking at the above one believe that the value of `element` is passed
in as first argument, and the the value of `element == "c"` as the
second, but that's not the case. It's much better if it's visible that
you got some kind of anonymous function definition there without
knowing about the "filter" built-in.

Java already has a syntax for expressing this kind of thing, so it
would be better to use that familiar syntax. As far as I know it
doesn't conflict with ours (it kind of it does, but not fatally).

Also, even if for now we only allow this in said built-ins, we
shouldn't exclude the possibility of making this kind of expression
accessible elsewhere as well. Then, on many places the parsed won't
know what kind of value is expected (consider passing a lambda to an
user defined directive for example), so a syntax like above won't
work.

> Not sure if that's what you have in mind too, but to me it made sense with
> regards to BuiltInsWithParseTimeParameters and I could start without
> touching parser stuff.

BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
bit...), as a call to a such built-in looks like a call to any other
built-in.

> *1st argument 'element'* would just be the iterator variable similar to
> <#list ["a","b","c"] as *element*>
> 2nd argument is the filter lambda expression... aka our filter condition
> 3rd+n argument are optional parameters in case used in the lambda expression

#list is special, similarly as `for ... in ...` is in most languages.
It's not lambda-ish either; you can't write `as element/2` or such.

> So at first I was looking how <#list> works and found IteratorBlock, and
> though I could reuse it somehow.
>
> Here is some simple pseudo code I played around for the for Example 1:
>
> static class filter_BI extends BuiltInWithParseTimeParameters {
>
>
>
>         TemplateModel _eval(Environment env) throws TemplateException {
>
>             // sequence
>
>         TemplateModel targetValue = target.evalToNonMissing(env);
>
>
>
>             List parameters = this.parameters;
>
>
>
>             Expression iteratorAlias = (Expression) parameters.get(0);
>
>             Expression conditionExpression = (Expression) parameters.get(1);
>
>
>
>             TemplateSequenceModel seq =  (TemplateSequenceModel) target.eval
> (env);
>
>             for (int i = 0; i < seq.size(); i++) {
>
>                TemplateModel cur = seq.get(i);
>
>
>                // this is where I am stuck at the moment
>
>                // I basically want to evaluate conditionExpression
> where iteratorAlias
> is basically what I passed as 'element'
>
>                // I am not sure if or how LocalContext could come into play
> here
>
>                // basically for each iteration I would assign the current
> loop element to a context variable with the name 'element'
>
>                // and then evaluate conditionExpression with that context.
>
>                // if conditionExpression is "true" then I would populate
> add the current sequence element 'cur'
>
>                // to a new result-List.... and return that.... something
>
>                // I wanted to reuse IteratorBlock here somehow, but didn't
> get it to work yet.
>
>                // maybe this is a stupid idea, or we just need something
> similar
>
>
>
>             }
>
> }
>
>
>
> Ok so far for my pseudo code....  Maybe you could give some more pointers
> based on that... in case this makes any sense ...

For LocalContext, Environment has a pushLocalContext method. See the
calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
Map, List).

To avoid running into more new things at once than necessary, perhaps
you should start with seq?seq_contains(lambda). The end result should
be like:

  users?contains(u -> u.paying)

>> Another similarity to BuiltInsWithParseTimeParameters is that we won't
>> allow separating the `?someBI` and `(arg)`. Like, with the example of
>> `cond?then(1, 2)`, you aren't allowed to do this:
>>
>>   <#assign t=cond?then>
>>   ${t(1, 2)}
>>
>> That maybe looks natural, but most other built-ins allow that. Of
>> course we can't allow that for ?filter etc., because then we need
>> closures again.
>>
>> >> Multi-part macro body is also planned. Means, I know it definitely
>> >> should be added, but who knows when that's done... I mean, it's like
>> >> that for what, a decade? (: It's not even decided what it exactly
>> >> does, as there are many ways of approaching this. (I have my own idea
>> >> about what the right compromise would be, but others has other
>> >> ideas...)
>> >>
>> >> Filtering lists bothers me because the template language should be
>> >> (and somewhat indeed is) specialized on listing things on fancy ways
>> >> that used to come up when generating document-like output. (If it
>> >> doesn't do things like that, you might as well use a general purpose
>> >> language.) Thus, that filter is unsolved (filtering with #if is
>> >> verbose and spoils #sep etc.) bothers me a lot.
>> >>
>> >> BTW, ?filter and ?map is also especially handy in our case as
>> >> FreeMarker doesn't support building new sequences (sequences are
>> >> immutable). Although it has sequence concatenation with `+`, it's not
>> >> good for building a sequence one by one, unless the sequence will be
>> >> quite short.
>> >>
>> > Good point.
>> >
>> >
>> >>
>> >> > Cheers,
>> >> > -- Denis.
>> >>
>> >> --
>> >> Thanks,
>> >>  Daniel Dekany
>> >>
>> >>
>> >
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
> Thanks
> Christoph
>

-- 
Thanks,
 Daniel Dekany

Re: Re : Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

Thanks Daniel for your feedback. See my answers below

Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <ddekany@apache.org
>:

> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>
> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
> ddekany@apache.org
> >>:
> >
> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
> >>
> >> > Hi,
> >> >
> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com> a
> >> écrit :
> >> >
> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
> >> ddekany@apache.org
> >> > :
> >> >
> >> > It's certainly tricky, but as far as I see possible (but then, who
> >> >
> >> > knows what will one find when actually working on it). It's also a
> >> > feature missing a lot. It's especially missing for #list (I know that
> >> > you need it for something else), because if you filter the items
> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be usable.
> >> >
> >> > Let me say that I disagree here.
> >> >
> >> > I do not think that closures are required for FreeMarker, nor that
> they
> >> are a good idea.
> >> >
> >> > If we add new features to the FreeMarker *tempate engine* I would
> >> > rather we focus on multi-part macro body rather than an advanced
> >> language feature like closures.
> >> >
> >> > You can add ?filter and ?map if you want, a simple expression as
> >> parameter should be enough.
> >>
> >> Yes, as I said, we certainly start with only allowing lambdas in
> >> ?filter/?map, also certainly in ?contains.
> >>
> > Would be enough in my opinion and very useful.
> >
> > Is it possiblefor you to give some pointers to the code on how this could
> > be implemented? I would maybe like to wrap my head around this a little
> bit.
>
> Please feel yourself encouraged! (:
>
> > I started looking at seq_containsBI (
> >
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
> )
> > and
> > and reverseBI (
> >
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
> )
> > just to find something related (seq_containsBI checks something) and
> > reverseBI returns a new sequence.
> > What I haven't found is a function which takes an Expression as a
> > parameter.
> > Is there something similar already or would that be a new thing?
>
> It's a new thing in that it will be part of the expression syntax
> (even if for now we will only allow lambdas as the parameters of a few
> built-ins, so that we can get away without closures). So it's a new
> Expression subclass, and has to be part of the parser (ftl.jj) as
> well.

Hmm, that parser stuff is new for me, it'll take me some time to get into
it.


> As of lazy evaluation of parameters expressions, that's already
> done in the built-ins in BuiltInsWithParseTimeParameters, and you will
> see it's trivial to do, but the situation there is much simpler.
>


>
> In principle, a LambdaExpression should evaluate to a
> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
> the called built-in or whatever it is. But with the approach of
> BuiltInsWithParseTimeParameters we can certainly even skip that, and
> just bind to the LambdaExpression directly, add a LocalContext that
> contains the lambda arguments, end evaluate the LambdaExpression right
> there, in the built-in implementation. Or at least at a very quick
> glance I think so.
>
Not sure I can follow completely but that hint with*
BuiltInsWithParseTimeParameters* got me started, but at the moment I'm
stuck as I need to get more familiar with the internals of Freemarker. I am
also not sure I am on the same page regarding the syntax we are aiming for
and why I would need to extend the parser when there is something like
BuiltInsWithParseTimeParameters....

Here is an example what I have in mind:
I started with the ?filter() builtin. I had a syntax like this in mind:

*Example 1: ["a","b","c"]?filter(element, element == "c")*
*Example 2: ["a","b","c"]?filter(element, element == someOtherVariable,
someOtherVariable="c")*
Not sure if that's what you have in mind too, but to me it made sense with
regards to BuiltInsWithParseTimeParameters and I could start without
touching parser stuff.

*1st argument 'element'* would just be the iterator variable similar to
<#list ["a","b","c"] as *element*>
2nd argument is the filter lambda expression... aka our filter condition
3rd+n argument are optional parameters in case used in the lambda expression

So at first I was looking how <#list> works and found IteratorBlock, and
though I could reuse it somehow.

Here is some simple pseudo code I played around for the for Example 1:

static class filter_BI extends BuiltInWithParseTimeParameters {



        TemplateModel _eval(Environment env) throws TemplateException {

            // sequence

        TemplateModel targetValue = target.evalToNonMissing(env);



            List parameters = this.parameters;



            Expression iteratorAlias = (Expression) parameters.get(0);

            Expression conditionExpression = (Expression) parameters.get(1);



            TemplateSequenceModel seq =  (TemplateSequenceModel) target.eval
(env);

            for (int i = 0; i < seq.size(); i++) {

               TemplateModel cur = seq.get(i);


               // this is where I am stuck at the moment

               // I basically want to evaluate conditionExpression
where iteratorAlias
is basically what I passed as 'element'

               // I am not sure if or how LocalContext could come into play
here

               // basically for each iteration I would assign the current
loop element to a context variable with the name 'element'

               // and then evaluate conditionExpression with that context.

               // if conditionExpression is "true" then I would populate
add the current sequence element 'cur'

               // to a new result-List.... and return that.... something

               // I wanted to reuse IteratorBlock here somehow, but didn't
get it to work yet.

               // maybe this is a stupid idea, or we just need something
similar



            }

}



Ok so far for my pseudo code....  Maybe you could give some more pointers
based on that... in case this makes any sense ...



Another similarity to BuiltInsWithParseTimeParameters is that we won't
> allow separating the `?someBI` and `(arg)`. Like, with the example of
> `cond?then(1, 2)`, you aren't allowed to do this:
>
>   <#assign t=cond?then>
>   ${t(1, 2)}
>
> That maybe looks natural, but most other built-ins allow that. Of
> course we can't allow that for ?filter etc., because then we need
> closures again.
>
> >> Multi-part macro body is also planned. Means, I know it definitely
> >> should be added, but who knows when that's done... I mean, it's like
> >> that for what, a decade? (: It's not even decided what it exactly
> >> does, as there are many ways of approaching this. (I have my own idea
> >> about what the right compromise would be, but others has other
> >> ideas...)
> >>
> >> Filtering lists bothers me because the template language should be
> >> (and somewhat indeed is) specialized on listing things on fancy ways
> >> that used to come up when generating document-like output. (If it
> >> doesn't do things like that, you might as well use a general purpose
> >> language.) Thus, that filter is unsolved (filtering with #if is
> >> verbose and spoils #sep etc.) bothers me a lot.
> >>
> >> BTW, ?filter and ?map is also especially handy in our case as
> >> FreeMarker doesn't support building new sequences (sequences are
> >> immutable). Although it has sequence concatenation with `+`, it's not
> >> good for building a sequence one by one, unless the sequence will be
> >> quite short.
> >>
> > Good point.
> >
> >
> >>
> >> > Cheers,
> >> > -- Denis.
> >>
> >> --
> >> Thanks,
> >>  Daniel Dekany
> >>
> >>
> >
>
> --
> Thanks,
>  Daniel Dekany
>
>
Thanks
Christoph

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 5596493Fax.: 
+49 3641 5596499
Internet: https://synesty.com <https://synesty.com>


Geschäftsführer: Christoph Rüger
Unternehmenssitz: Jena
Handelsregister B 
beim Amtsgericht: Jena
Handelsregister-Nummer: HRB 508766
Ust-IdNr.: 
DE287564982

Re: Re : Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:

> Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <ddekany@apache.org
>>:
>
>> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>>
>> > Hi,
>> >
>> > Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com> a
>> écrit :
>> >
>> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>> ddekany@apache.org
>> > :
>> >
>> > It's certainly tricky, but as far as I see possible (but then, who
>> >
>> > knows what will one find when actually working on it). It's also a
>> > feature missing a lot. It's especially missing for #list (I know that
>> > you need it for something else), because if you filter the items
>> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be usable.
>> >
>> > Let me say that I disagree here.
>> >
>> > I do not think that closures are required for FreeMarker, nor that they
>> are a good idea.
>> >
>> > If we add new features to the FreeMarker *tempate engine* I would
>> > rather we focus on multi-part macro body rather than an advanced
>> language feature like closures.
>> >
>> > You can add ?filter and ?map if you want, a simple expression as
>> parameter should be enough.
>>
>> Yes, as I said, we certainly start with only allowing lambdas in
>> ?filter/?map, also certainly in ?contains.
>>
> Would be enough in my opinion and very useful.
>
> Is it possiblefor you to give some pointers to the code on how this could
> be implemented? I would maybe like to wrap my head around this a little bit.

Please feel yourself encouraged! (:

> I started looking at seq_containsBI (
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291)
> and
> and reverseBI (
> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264)
> just to find something related (seq_containsBI checks something) and
> reverseBI returns a new sequence.
> What I haven't found is a function which takes an Expression as a
> parameter.
> Is there something similar already or would that be a new thing?

It's a new thing in that it will be part of the expression syntax
(even if for now we will only allow lambdas as the parameters of a few
built-ins, so that we can get away without closures). So it's a new
Expression subclass, and has to be part of the parser (ftl.jj) as
well. As of lazy evaluation of parameters expressions, that's already
done in the built-ins in BuiltInsWithParseTimeParameters, and you will
see it's trivial to do, but the situation there is much simpler.

In principle, a LambdaExpression should evaluate to a
TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
the called built-in or whatever it is. But with the approach of
BuiltInsWithParseTimeParameters we can certainly even skip that, and
just bind to the LambdaExpression directly, add a LocalContext that
contains the lambda arguments, end evaluate the LambdaExpression right
there, in the built-in implementation. Or at least at a very quick
glance I think so.

Another similarity to BuiltInsWithParseTimeParameters is that we won't
allow separating the `?someBI` and `(arg)`. Like, with the example of
`cond?then(1, 2)`, you aren't allowed to do this:

  <#assign t=cond?then>
  ${t(1, 2)}

That maybe looks natural, but most other built-ins allow that. Of
course we can't allow that for ?filter etc., because then we need
closures again.

>> Multi-part macro body is also planned. Means, I know it definitely
>> should be added, but who knows when that's done... I mean, it's like
>> that for what, a decade? (: It's not even decided what it exactly
>> does, as there are many ways of approaching this. (I have my own idea
>> about what the right compromise would be, but others has other
>> ideas...)
>>
>> Filtering lists bothers me because the template language should be
>> (and somewhat indeed is) specialized on listing things on fancy ways
>> that used to come up when generating document-like output. (If it
>> doesn't do things like that, you might as well use a general purpose
>> language.) Thus, that filter is unsolved (filtering with #if is
>> verbose and spoils #sep etc.) bothers me a lot.
>>
>> BTW, ?filter and ?map is also especially handy in our case as
>> FreeMarker doesn't support building new sequences (sequences are
>> immutable). Although it has sequence concatenation with `+`, it's not
>> good for building a sequence one by one, unless the sequence will be
>> quite short.
>>
> Good point.
>
>
>>
>> > Cheers,
>> > -- Denis.
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
>

-- 
Thanks,
 Daniel Dekany

Re: Re : Re: Lambda Expressions - filter list without <#list> directive

[Christoph Rüger <c....@synesty.com>]

Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <ddekany@apache.org
>:

> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>
> > Hi,
> >
> > Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com> a
> écrit :
> >
> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
> ddekany@apache.org
> > :
> >
> > It's certainly tricky, but as far as I see possible (but then, who
> >
> > knows what will one find when actually working on it). It's also a
> > feature missing a lot. It's especially missing for #list (I know that
> > you need it for something else), because if you filter the items
> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be usable.
> >
> > Let me say that I disagree here.
> >
> > I do not think that closures are required for FreeMarker, nor that they
> are a good idea.
> >
> > If we add new features to the FreeMarker *tempate engine* I would
> > rather we focus on multi-part macro body rather than an advanced
> language feature like closures.
> >
> > You can add ?filter and ?map if you want, a simple expression as
> parameter should be enough.
>
> Yes, as I said, we certainly start with only allowing lambdas in
> ?filter/?map, also certainly in ?contains.
>
Would be enough in my opinion and very useful.

Is it possiblefor you to give some pointers to the code on how this could
be implemented? I would maybe like to wrap my head around this a little bit.
I started looking at seq_containsBI (
https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291)
and
and reverseBI (
https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264)
just to find something related (seq_containsBI checks something) and
reverseBI returns a new sequence.
What I haven't found is a function which takes an Expression as a
parameter.
Is there something similar already or would that be a new thing?


>
> Multi-part macro body is also planned. Means, I know it definitely
> should be added, but who knows when that's done... I mean, it's like
> that for what, a decade? (: It's not even decided what it exactly
> does, as there are many ways of approaching this. (I have my own idea
> about what the right compromise would be, but others has other
> ideas...)
>
> Filtering lists bothers me because the template language should be
> (and somewhat indeed is) specialized on listing things on fancy ways
> that used to come up when generating document-like output. (If it
> doesn't do things like that, you might as well use a general purpose
> language.) Thus, that filter is unsolved (filtering with #if is
> verbose and spoils #sep etc.) bothers me a lot.
>
> BTW, ?filter and ?map is also especially handy in our case as
> FreeMarker doesn't support building new sequences (sequences are
> immutable). Although it has sequence concatenation with `+`, it's not
> good for building a sequence one by one, unless the sequence will be
> quite short.
>
Good point.


>
> > Cheers,
> > -- Denis.
>
> --
> Thanks,
>  Daniel Dekany
>
>

-- 
Synesty GmbH
Moritz-von-Rohr-Str. 1a
07745 Jena
Tel.: +49 3641 5596493Fax.: 
+49 3641 5596499
Internet: https://synesty.com <https://synesty.com>


Geschäftsführer: Christoph Rüger
Unternehmenssitz: Jena
Handelsregister B 
beim Amtsgericht: Jena
Handelsregister-Nummer: HRB 508766
Ust-IdNr.: 
DE287564982

Re: Re : Re: Lambda Expressions - filter list without <#list> directive

[Daniel Dekany <dd...@apache.org>]

Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:

> Hi,
>
> Le 9 novembre 2018 à 22:36, Christoph Rüger <c....@synesty.com> a écrit :
>
> Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <ddekany@apache.org
> :
>
> It's certainly tricky, but as far as I see possible (but then, who
>
> knows what will one find when actually working on it). It's also a
> feature missing a lot. It's especially missing for #list (I know that
> you need it for something else), because if you filter the items
> inside #list with #if-s, then #sep, ?hasNext, etc. will not be usable.
>
> Let me say that I disagree here.
>
> I do not think that closures are required for FreeMarker, nor that they are a good idea.
>
> If we add new features to the FreeMarker *tempate engine* I would
> rather we focus on multi-part macro body rather than an advanced language feature like closures.
>
> You can add ?filter and ?map if you want, a simple expression as parameter should be enough.

Yes, as I said, we certainly start with only allowing lambdas in
?filter/?map, also certainly in ?contains.

Multi-part macro body is also planned. Means, I know it definitely
should be added, but who knows when that's done... I mean, it's like
that for what, a decade? (: It's not even decided what it exactly
does, as there are many ways of approaching this. (I have my own idea
about what the right compromise would be, but others has other
ideas...)

Filtering lists bothers me because the template language should be
(and somewhat indeed is) specialized on listing things on fancy ways
that used to come up when generating document-like output. (If it
doesn't do things like that, you might as well use a general purpose
language.) Thus, that filter is unsolved (filtering with #if is
verbose and spoils #sep etc.) bothers me a lot.

BTW, ?filter and ?map is also especially handy in our case as
FreeMarker doesn't support building new sequences (sequences are
immutable). Although it has sequence concatenation with `+`, it's not
good for building a sequence one by one, unless the sequence will be
quite short.

> Cheers,
> -- Denis.

-- 
Thanks,
 Daniel Dekany