You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Do...@swisscom.com on 2026/08/07 09:31:25 UTC
Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER)
Hi Gustavo,
Sorry for the late response. I had "a few" discussions with my colleague Michal Stutzmann (you might remember him from the Flink Forward conference in Barcelona) about the FLIP and the features it should cover. Michal extended the proposal<https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing> and implementation to also support the Table & Python APIs, as well as lambda arguments for user-defined functions. Additionally, we now support using columns within lambda expressions.
We pushed the WIP changes to a branch in our fork, which you can find here:
https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas
The set of changes grew larger than expected, but in our opinion they significantly simplify the user experience by reducing the need to work with UDFs or custom code to handle collections.
Thanks a lot in advance for taking a look!
Best,
Dominik
From: Gustavo de Morais <gu...@gmail.com>
Date: Friday, 31 July 2026 at 12:05
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER)
Be aware: This is an external email.
Hey Dominik,
Lambda functions are an important building block - not only for these
built-in collection functions, but also for UDFs and PTFs whitin the whole
archictecture. So thanks for the proposal and for wanting to contribute.
I'll take a look at the proposal soon. You mentioned a WIP branch a couple
of times in the thread - could you share the link so we can take a look at
that too?
Kind regards,
Gustavo
On Thu, 30 Jul 2026 at 16:39, <Do...@swisscom.com> wrote:
> Hi all,
>
> Following up on this FLIP to see if there are any comments or concerns.
>
> Since the last mail I added tests for:
> - views using ARRAY_FILTER / TRANSFORM
> - materialized tables going through expand-and-replan
>
> Everything is green locally so far.
>
> Feedback on the planner integration, Calcite handling, missing edge cases,
> or the overall direction would be appreciated.
>
> As I’m still new to the FLIP process, I’d also appreciate some guidance on
> the next steps from here. Many thanks in advance.
>
> Best,
> Dominik
>
> From: Bünzli Dominik, SCS-INI-DNA-INF <Do...@swisscom.com>
> Date: Friday, 24 July 2026 at 11:17
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> (TRANSFORM and ARRAY_FILTER)
>
> Hi Sergey,
>
> Good catch. I didn’t have these cases in mind. Quickly added test cases
> for them to my branch.
>
> The view test creates a view like "CREATE VIEW v AS SELECT id,
> ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and checks
> that the arrays come back right. Same idea for TRANSFORM and the map lambda
> (k, v) -> v * 100.
>
> The materialized table test does the equivalent: a CONTINUOUS table using
> ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, then a
> SELECT to confirm the output matches.
>
> So, both should (to my still limited knowledge), go through the
> expand-and-replan path and check real results. If Calcite mangles a lambda,
> these tests should fail. Looks good so far, tests are green locally.
>
> Best,
> Dominik
>
> From: Sergey Nuyanzin <sn...@gmail.com>
> Date: Friday, 24 July 2026 at 09:59
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> (TRANSFORM and ARRAY_FILTER)
>
>
> Be aware: This is an external email.
>
>
>
> thanks for the proposal
>
> and working in this direction
>
> I haven't checked yet all the things
>
> however I see at least one missing piece here.
>
> In FlinkSQL we have views and materialized tables. Both are relying on
> expanded ("rewritten" in Calcite terms) SQL.
> And we have already faced a number of issues around that and fixed
> them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6],
> [7], [8], [9], [10])
>
> I'm pretty sure there will be something lambda related. For that
> reason I would suggest also having test with views(or materialized
> tables, does not matter) based on sql with lambda
> and check that SELECT from them still able to produce output
>
> [1]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660>>
> [2]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575>>
> [3]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465>>
> [4]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480>>
> [5]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471>>
> [6]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470>>
> [7]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466>>
> [8]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312>>
> [9]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7217>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7217>>
> [10]
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944>
> <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944>>
>
> On Fri, Jul 24, 2026 at 9:37 AM <Do...@swisscom.com> wrote:
> >
> > Hi Gyula
> >
> > Thanks a lot for your reply and the +1!
> >
> > On a conceptual level, lambdas and scalar functions / UDFs are distinct.
> A lambda isn't a stored/registered value. It's represented by the
> planning-only FUNCTION logical type and only exists as an argument to
> higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially, it's
> the wrapper that introduces the element variable x.
> >
> > Within the lambda you can freely use any scalar function or UDF:
> > ARRAY_FILTER(arr, x -> is_positive(x))
> > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100)
> >
> > What currently doesn't work is passing a bare UDF name in the lambda
> position (i.e. automatic expansion of is_positive into x -> is_positive(x)):
> > ARRAY_FILTER(arr, is_positive) -- not supported
> >
> > We intentionally left this out of scope for now, but it's a
> straightforward follow-up if we see demand for it.
> >
> > Best,
> > Dominik
> >
> > From: Gyula Fóra <gy...@gmail.com>
> > Date: Thursday, 23 July 2026 at 16:07
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> (TRANSFORM and ARRAY_FILTER)
> >
> >
> > Be aware: This is an external email.
> >
> >
> >
> > Hey!
> >
> > Without a good background knowledge about the syntax in other systems,
> the
> > ARRAY_FILTER, TRANSFORM functions seem really useful, especially with
> > lambda functions.
> > Overall +1 for this idea from a very high level but it would be great
> with
> > someone with more deep context on SQL functions / runtime to chime in.
> >
> > One question I had is how do lambda functions relate to existing scalar
> > functions / UDFs? Would the user be able to use boolean valued scalar
> > functions where lambdas are shown in the proposal?
> >
> > Cheers
> > Gyula
> >
> > On Wed, Jul 15, 2026 at 8:40 AM <Do...@swisscom.com> wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to start a discussion on a FLIP that introduces higher-order
> > > functions
> > > (functions that take a lambda expression as an argument) to Flink SQL,
> > > together
> > > with the first two built-ins that use them: TRANSFORM and ARRAY_FILTER.
> > >
> > > FLIP:
> > >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> ><
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> >
> > >
> > > Motivation
> > >
> > > Users migrating to Flink SQL from Spark, Databricks, Snowflake,
> DuckDB, and
> > > Presto/Trino expect to manipulate collections (arrays and maps) inline
> > > with a
> > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. The
> > > absence of
> > > higher-order collection functions forces verbose query rewrites during
> > > migration
> > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and
> > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry points
> > > and,
> > > importantly, they can be built on lambda infrastructure that already
> > > exists in
> > > Calcite (CALCITE-3679), so the surface area we add on the Flink side is
> > > rather
> > > small.
> > >
> > > Summary
> > >
> > > • Introduce a new logical type FUNCTION that describes the type of a
> > > lambda
> > > (its argument types and its result type). This is the type-system
> > > foundation
> > > every higher-order function needs; it is a planning/translation
> helper
> > > type
> > > and is not a persisted column type.
> > > • Add ARRAY_FILTER(array, element -> predicate), which returns a new
> > > array
> > > containing only the elements for which the predicate holds.
> > > • Add TRANSFORM(collection, lambda), which applies a lambda to every
> > > element of
> > > an array (TRANSFORM(array, x -> expr)) or every entry of a map
> > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map.
> > > • Reuse Calcite's lambda parsing, validation and
> > > RexLambda/FunctionSqlType
> > > infrastructure (CALCITE-3679) rather than inventing a
> Flink-specific
> > > lambda
> > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is
> already
> > > parseable by the Calcite version Flink bundles (1.41.0).
> > >
> > > The functions are net-new syntax and are additive: no existing query
> > > changes
> > > behavior. There is no new configuration option — the functions are
> always
> > > available once the release ships.
> > >
> > > Examples:
> > >
> > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); --
> > > [3, 4]
> > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); --
> > > [10, 20, 30]
> > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s)); --
> > > [1, 2, 3]
> > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); --
> > > {a=100, b=200}
> > >
> > > Thanks,
> > > Dominik Bünzli
> > > Data, Analytics & AI Engineer
> > >
>
>
>
> --
> Best regards,
> Sergey
>
Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER)
Posted by Do...@swisscom.com.
Hi Martijn
Yes, Michal, Lorand and me are continuing with the work. We’re currently in the process to split, simplify and review the FLIP to make it more readable.
Best,
Dominik
From: Martijn Visser <ma...@apache.org>
Date: Thursday, 17 September 2026 at 19:35
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER)
Be aware: This is an external email.
Hi all,
I was wondering if this proposal is still moving forward? I'd love to
see some of these changes land in the upcoming Flink release.
Thanks, Martijn
Op vr 21 aug 2026 om 20:27 schreef David Anderson <da...@apache.org>:
>
> +1 for the overall direction; we've needed this. It will significantly
> improve both usability and readability.
>
> Also +1 for Gustavo's recommendations. The overall scope is just too much
> for one FLIP.
>
> On Fri, Aug 21, 2026 at 8:42 AM Gustavo de Morais <gu...@gmail.com>
> wrote:
>
> > Hi Michal and Dominik,
> >
> > Thanks for working on this - the new document is much more mature than the
> > first one! In general, I think we're in a good direction, +1.
> >
> > A few points:
> >
> > 1) Splitting the FLIP
> >
> > This is a lot of content for careful review, and each function really needs
> > its own discussion. If we include so many in one FLIP, the review will
> > probably take even longer. I'd suggest we focus on the foundation first and
> > get that cleared, and have the function discussions in a second FLIP (and
> > partially on the pull requests).
> >
> > Suggestion:
> > - FLIP 1 (foundation): lambda syntax + FUNCTION type + capture lifting +
> > codegen + Table API/Python etc. It can include a few example functions
> > (e.g. ARRAY_TRANSFORM, ARRAY_FILTER, ARRAY_REDUCE) to exercise the design,
> > but without committing to their shape.
> > - FLIP 2+ (breadth): a second FLIP to discuss the first set of functions,
> > their APIs and shapes.
> >
> > 2) Making it a FLIP document
> >
> > The document currently reads like a long decision log. To make it official
> > as a FLIP, I think we should turn it into a concise, easy-to-read document:
> > - Make it as short as possible with ofc still all the relevant information
> > we need. Right now at 85 pages, it's really only consumable by AI.
> > - Add an index at the top so it's easy to navigate. We often come back to a
> > FLIP many times over the years, so this matters.
> > - Add examples wherever we can - examples are in general super helpful to
> > immediately grasp the content. There are some short ones in the
> > "Composition" section - add more whetever you can.
> > - As orientation, it helps to look at how recent FLIPs are structured. I'm
> > happy to help make sure this lands on our Confluence when we're ready.
> >
> > 3) Content feedback (for later, not the first FLIP)
> >
> > Note: I don't think these belong in the first FLIP we should focus on, so
> > feel free to ignore them for now - just sharing for the future and feel
> > free to not reply directly to these here:
> > - Predicates like exists/forall (Spark) and any_match/all_match/none_match
> > (Trino). Very common, and hard to express well with FILTER + CARDINALITY.
> > - An (element, index) variant of transform/filter, like Spark/Trino
> > transform(a, (x, i) -> ...) with i the 0-based index. Pretty core and
> > useful.
> > - A custom comparator array_sort(array, comparator), as in Spark/Trino
> > (2-arg comparator returning negative/0/positive).
> >
> > 4) EXPLAIN improvement
> >
> > For a capturing lambda, the plan shows the lifted form: the user writes x
> > -> x + base, but the plan renders (x, cap$1) -> (x + cap$1), base. Could we
> > de-lift at render time so EXPLAIN shows intent, not the optimization - i.e.
> > Calc(select=[ARRAY_TRANSFORM(a, x -> (x + base)) AS EXPR$0])? The internal
> > representation can stay as-is.
> >
> > Thanks again for driving this,
> >
> > Kind regards,
> > Gustavo
> >
> > On Sun, 9 Aug 2026 at 10:05, Michal Stutzmann <mi...@gmail.com>
> > wrote:
> >
> > > Hi Gustavo,
> > >
> > > Thanks for looking into this and the pointers. We have now included more
> > > details in the FLIP draft (link shared by Dominik in the previous e-mail:
> > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633540880143%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=W0tuQVrQiCfZaYKATeh4NVPHyZ4TcWhyhKMw8T77ny8%3D&reserved=0<https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing>
> > > ).
> > >
> > > In short:
> > > - A UDF called from a lambda body — supported.
> > > - A UDF passed as the lambda, e.g. `ARRAY_TRANSFORM(a, my_udf)`, is not
> > > supported — this would need eta-expansion. The same call written as
> > > `ARRAY_TRANSFORM(a, x -> my_udf(x))` is supported.
> > > - Nesting: supported; arbitrary depth, any combination of built-in and
> > > user-defined, in both directions, with each level capturing the
> > > parameters of all enclosing lambdas plus outer columns.
> > >
> > > All three are covered by a new section in the FLIP draft called Public
> > > Interfaces / 6. Composition, which lists the supported compositions with
> > an
> > > example for each. The mechanism behind it is in Proposed Changes / Lambda
> > > captures.
> > >
> > > Every one of those is covered by an IT case that executes the query and
> > > asserts the result. If you have other combinations in mind, we will add
> > > them.
> > >
> > > Best,
> > > Michal
> > >
> > >
> > > On Fri, Aug 7, 2026 at 4:15 PM Gustavo de Morais <gustavopguto@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Dominik,
> > > >
> > > > Thanks for the reply and for considering these. It's great that you're
> > > both
> > > > working on this. As mentioned, it's a very important feature, but there
> > > are
> > > > indeed some cases that might grow its scope - we should make sure we
> > > design
> > > > it in a complete way that fits the different parts of the project.
> > > >
> > > > I'll try to make some time next week to take a deeper look at what you
> > > > shared. In the meantime, here are a few more pointers you can already
> > > look
> > > > into: you mentioned lambda arguments for user-defined functions - what
> > > > about a UDF as an argument of a lambda function? Nesting is also worth
> > > > considering: multiple nested lambda functions, lambda function -> UDF
> > or
> > > > regular function -> lambda function -> ... etc.
> > > >
> > > > Kind regards,
> > > > Gustavo
> > > >
> > > > On Fri, 7 Aug 2026 at 11:31, <Do...@swisscom.com> wrote:
> > > >
> > > > > Hi Gustavo,
> > > > >
> > > > > Sorry for the late response. I had "a few" discussions with my
> > > colleague
> > > > > Michal Stutzmann (you might remember him from the Flink Forward
> > > > conference
> > > > > in Barcelona) about the FLIP and the features it should cover. Michal
> > > > > extended the proposal<
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633540929556%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=wEatAvBZAX%2BEgX0GRb2Aof9ygPqQDSbo0Z0rXIJZUGI%3D&reserved=0<https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing>
> > > > >
> > > > > and implementation to also support the Table & Python APIs, as well
> > as
> > > > > lambda arguments for user-defined functions. Additionally, we now
> > > support
> > > > > using columns within lambda expressions.
> > > > >
> > > > > We pushed the WIP changes to a branch in our fork, which you can find
> > > > here:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fswisscom-bigdata%2Fflink%2Ftree%2Fflink-31207-higher-order-functions-and-lambdas&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633540975518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2BXaXfJdoa6Cf4d707fEEnQpxsThqbR4GcR4tTpeQOfM%3D&reserved=0<https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas>
> > > > >
> > > > > The set of changes grew larger than expected, but in our opinion they
> > > > > significantly simplify the user experience by reducing the need to
> > work
> > > > > with UDFs or custom code to handle collections.
> > > > >
> > > > > Thanks a lot in advance for taking a look!
> > > > >
> > > > > Best,
> > > > > Dominik
> > > > >
> > > > > From: Gustavo de Morais <gu...@gmail.com>
> > > > > Date: Friday, 31 July 2026 at 12:05
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > > > > (TRANSFORM and ARRAY_FILTER)
> > > > >
> > > > >
> > > > > Be aware: This is an external email.
> > > > >
> > > > >
> > > > >
> > > > > Hey Dominik,
> > > > >
> > > > > Lambda functions are an important building block - not only for these
> > > > > built-in collection functions, but also for UDFs and PTFs whitin the
> > > > whole
> > > > > archictecture. So thanks for the proposal and for wanting to
> > > contribute.
> > > > >
> > > > > I'll take a look at the proposal soon. You mentioned a WIP branch a
> > > > couple
> > > > > of times in the thread - could you share the link so we can take a
> > look
> > > > at
> > > > > that too?
> > > > >
> > > > > Kind regards,
> > > > > Gustavo
> > > > >
> > > > > On Thu, 30 Jul 2026 at 16:39, <Do...@swisscom.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Following up on this FLIP to see if there are any comments or
> > > concerns.
> > > > > >
> > > > > > Since the last mail I added tests for:
> > > > > > - views using ARRAY_FILTER / TRANSFORM
> > > > > > - materialized tables going through expand-and-replan
> > > > > >
> > > > > > Everything is green locally so far.
> > > > > >
> > > > > > Feedback on the planner integration, Calcite handling, missing edge
> > > > > cases,
> > > > > > or the overall direction would be appreciated.
> > > > > >
> > > > > > As I’m still new to the FLIP process, I’d also appreciate some
> > > guidance
> > > > > on
> > > > > > the next steps from here. Many thanks in advance.
> > > > > >
> > > > > > Best,
> > > > > > Dominik
> > > > > >
> > > > > > From: Bünzli Dominik, SCS-INI-DNA-INF <
> > Dominik.Buenzli@swisscom.com>
> > > > > > Date: Friday, 24 July 2026 at 11:17
> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > SQL
> > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > >
> > > > > > Hi Sergey,
> > > > > >
> > > > > > Good catch. I didn’t have these cases in mind. Quickly added test
> > > cases
> > > > > > for them to my branch.
> > > > > >
> > > > > > The view test creates a view like "CREATE VIEW v AS SELECT id,
> > > > > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and
> > > > checks
> > > > > > that the arrays come back right. Same idea for TRANSFORM and the
> > map
> > > > > lambda
> > > > > > (k, v) -> v * 100.
> > > > > >
> > > > > > The materialized table test does the equivalent: a CONTINUOUS table
> > > > using
> > > > > > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster,
> > > > then
> > > > > a
> > > > > > SELECT to confirm the output matches.
> > > > > >
> > > > > > So, both should (to my still limited knowledge), go through the
> > > > > > expand-and-replan path and check real results. If Calcite mangles a
> > > > > lambda,
> > > > > > these tests should fail. Looks good so far, tests are green
> > locally.
> > > > > >
> > > > > > Best,
> > > > > > Dominik
> > > > > >
> > > > > > From: Sergey Nuyanzin <sn...@gmail.com>
> > > > > > Date: Friday, 24 July 2026 at 09:59
> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > SQL
> > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > >
> > > > > >
> > > > > > Be aware: This is an external email.
> > > > > >
> > > > > >
> > > > > >
> > > > > > thanks for the proposal
> > > > > >
> > > > > > and working in this direction
> > > > > >
> > > > > > I haven't checked yet all the things
> > > > > >
> > > > > > however I see at least one missing piece here.
> > > > > >
> > > > > > In FlinkSQL we have views and materialized tables. Both are relying
> > > on
> > > > > > expanded ("rewritten" in Calcite terms) SQL.
> > > > > > And we have already faced a number of issues around that and fixed
> > > > > > them in Calcite (like just a few or them [1], [2], [3], [4], [5],
> > > [6],
> > > > > > [7], [8], [9], [10])
> > > > > >
> > > > > > I'm pretty sure there will be something lambda related. For that
> > > > > > reason I would suggest also having test with views(or materialized
> > > > > > tables, does not matter) based on sql with lambda
> > > > > > and check that SELECT from them still able to produce output
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541020222%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=23VQsbtHmdqiQMDleKDQwEt%2B2BVZ7vnlc3AidD9UevY%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541061340%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=SO0FuXgJDlGXmugXhi6DeXgLX3zn%2Bc%2BbrGd6Dh4qJCs%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660>>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541105102%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=I%2BtIVgqyDFEJhzKD4lfAwgp8z%2BO0phiA%2FsAgz%2BN5T0Q%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541152407%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0ChQR7tLLJdv4%2FClr5qIWwgrLjd%2F4cB4qG8rcA3R94Q%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660>>>
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541204097%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JMxWbF1iIboDm2kteytHau7DVioMV64eKTjAi5pVSN4%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541233971%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=91yGAtz9ATpy0RJEPyWmeSi8G3OdlOYlkYRewVq9Q7c%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575>>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541276521%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=hDIfpmBO1IE2uTNnamqSja4LfsDdrbFEz14Yn%2FG0oZc%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541311043%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=hWSMB7A1tJu04Dwj3kpkgtXcilKghfAYqzOGuAj1nZQ%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575>>>
> > > > > > [3]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541340378%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EompzpU3BbR%2FSytojMf0RP8zbHTjYZJ7KdVEFVYZ1LY%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541366885%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=oiqvLE29If6pNsiXDN1aMBVXd1wCwmPKnckO38UecbI%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465>>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541398942%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2BbeXZLiSL6kjQ6rhaWdjZ0YELv%2Bv16cJtN0OkFNrP0%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541424927%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8NDu6%2FrRVaabGi3vS3faxvsZpZBskAUNj5ABFzChDBo%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465>>>
> > > > > > [4]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541453581%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=G5%2Bw5wtzSiIji6ZqIP0ob%2FqBfSTZooo1pPmkdnsPazo%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541490184%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=NrMhVVzp9ZEgRsCTMyeETIGQIF2s4whwcxzAPr5230s%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480>>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541521366%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WE5pb%2Bt1Gczq0qrGW%2BsG3AMV9VPAm4D4BF90khadv4I%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541550720%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=YKPeJOfbfrXGMqv1uiuHlZI63HitM5vbnDevWBtF2FM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480>>>
> > > > > > [5]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541578307%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ppw%2BbUt2cmFOBxFEMFYK4Siws5TEz7qF%2B3r5wAkiJ5U%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541603974%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Vn8C1d%2FQE8KQi1qB5rIWTet6m9zVqh1gQp01TB9yvik%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471>>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541628656%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=KLS%2FI8y6c1W1uZhu4yDR7YaMDKkdiMQC9CPv8BKTHo8%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541653276%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9vbhBEffowsYVsGY2NoR1v6d4MaKRJV0NXpGE3790lQ%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471>>>
> > > > > > [6]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541677620%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=YucR3aRJ6VxBQ%2B4SQBfYn0i6AH2CpfoxWqeDGaTI45I%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541704487%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S%2BXgaG4eqo99Cq5pe4kJ7DWtGcURN6%2BBEITNw4gw8kM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470>>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541732358%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=zSP8KOFQuDoV3wq8mk1c9DHyROBN%2FULW2udr5w40XWw%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541776515%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=oF017QAs2V30E8qyOco9d%2BILvCHygv6aoRcO6CrL6iU%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470>>>
> > > > > > [7]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541833014%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=1u2gdozNrSTawlrbJ%2BFOYhXm%2BtImO%2BldEKf45w9yf6I%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541962220%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kvMrs0ah6zr%2FsfBbs2lwDx5gmLY0fcyJeg7mwjs0kyA%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466>>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633541998512%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EBL3To%2Bbif0kjHv7qnT2RL6J771nN2TogqopUIoldkg%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542036034%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=57WI8ZQSAv%2FvnRpOtbGEIoZ4bKBPvH%2BgAm9cgucnPI0%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466>>>
> > > > > > [8]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542076810%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kknmdzIX6u%2B96Qbdh6Q11JssrspiCunMOsm2rcXovdo%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542112448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=pPl34LC1vkV3X7n4zwSfgKCixdAFBvx7mc%2FaAoi6LDE%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312>>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542146125%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=A7Fjbzui3syCRX%2BlIQJJm5p1IN227fE97MD2bak43wM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542198137%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=uvQHb3MdccB9ROuHpQiSpQdbBUAS5HA98o3tQEm87RE%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312>>>
> > > > > > [9]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542248568%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XAetzHUvL%2F1egI2gguUOk3y1N6elrhoE2Se2u1utrrQ%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7217>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542298450%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=VrXOLxPrivewNutLfiFzMwo85lyN7hYMM48MHjDgkvU%3D&reserved=0>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542360486%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=9TRbtm2GnAuLpfXTfLFG8LqnFHwUyg0XXm3UjRYQkz8%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7217>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542410879%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=I9EKzPqedzrvwT1mNb6wVYF9dYaUyu707vLDRaiFFms%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7217>>>
> > > > > > [10]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542471894%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=821BEPtfj0dn9JPGozTu6BiwEZ66Q8duIoIBmDg%2BfqM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542535940%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=5P09o1lZfuyw7dsCe834ALFsM7mx1eBsbc9a1JoZxbI%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944>>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542579737%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=QqR3yJzTLYesFBAqIkIK1Q5Or4iUX01%2F2UVPsr8aXvc%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944>
> > > > > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542620076%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=bJpkyz%2BUSnty3zvjjeYioM8%2Fyyi2J%2F7%2B4jx4m31kVKA%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944>>>
> > > > > >
> > > > > > On Fri, Jul 24, 2026 at 9:37 AM <Do...@swisscom.com>
> > > wrote:
> > > > > > >
> > > > > > > Hi Gyula
> > > > > > >
> > > > > > > Thanks a lot for your reply and the +1!
> > > > > > >
> > > > > > > On a conceptual level, lambdas and scalar functions / UDFs are
> > > > > distinct.
> > > > > > A lambda isn't a stored/registered value. It's represented by the
> > > > > > planning-only FUNCTION logical type and only exists as an argument
> > to
> > > > > > higher-order functions like ARRAY_FILTER and TRANSFORM.
> > Essentially,
> > > > it's
> > > > > > the wrapper that introduces the element variable x.
> > > > > > >
> > > > > > > Within the lambda you can freely use any scalar function or UDF:
> > > > > > > ARRAY_FILTER(arr, x -> is_positive(x))
> > > > > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100)
> > > > > > >
> > > > > > > What currently doesn't work is passing a bare UDF name in the
> > > lambda
> > > > > > position (i.e. automatic expansion of is_positive into x ->
> > > > > is_positive(x)):
> > > > > > > ARRAY_FILTER(arr, is_positive) -- not supported
> > > > > > >
> > > > > > > We intentionally left this out of scope for now, but it's a
> > > > > > straightforward follow-up if we see demand for it.
> > > > > > >
> > > > > > > Best,
> > > > > > > Dominik
> > > > > > >
> > > > > > > From: Gyula Fóra <gy...@gmail.com>
> > > > > > > Date: Thursday, 23 July 2026 at 16:07
> > > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > > SQL
> > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > > >
> > > > > > >
> > > > > > > Be aware: This is an external email.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hey!
> > > > > > >
> > > > > > > Without a good background knowledge about the syntax in other
> > > > systems,
> > > > > > the
> > > > > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially
> > > with
> > > > > > > lambda functions.
> > > > > > > Overall +1 for this idea from a very high level but it would be
> > > great
> > > > > > with
> > > > > > > someone with more deep context on SQL functions / runtime to
> > chime
> > > > in.
> > > > > > >
> > > > > > > One question I had is how do lambda functions relate to existing
> > > > scalar
> > > > > > > functions / UDFs? Would the user be able to use boolean valued
> > > scalar
> > > > > > > functions where lambdas are shown in the proposal?
> > > > > > >
> > > > > > > Cheers
> > > > > > > Gyula
> > > > > > >
> > > > > > > On Wed, Jul 15, 2026 at 8:40 AM <Do...@swisscom.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I'd like to start a discussion on a FLIP that introduces
> > > > higher-order
> > > > > > > > functions
> > > > > > > > (functions that take a lambda expression as an argument) to
> > Flink
> > > > > SQL,
> > > > > > > > together
> > > > > > > > with the first two built-ins that use them: TRANSFORM and
> > > > > ARRAY_FILTER.
> > > > > > > >
> > > > > > > > FLIP:
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542662520%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ukZgFlO8yxEX13TrQxLOzVvWQeKYGz%2Fj%2BW2K6CChCck%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542705910%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=qb8S6soK8vJLV0YYXtewRRcKuwN%2FnZEy1hGJnvCWwE8%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> > > > > >
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542740491%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Blq8TrhpIRCU6%2Fg1J4csMnku6idl1Xo15h35O3uva4w%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542780849%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=RhW0%2FeqtahQphc7rtf8mXvAp9U9r0US016R%2BvwjUvnc%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> > > > > >
> > > > > > ><
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542825122%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gZEEBz5CPYkKDdR65XMCX%2BvXnvdWMedSUoNLU9UJdNg%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C777520c33f7145ab636408df14e21c72%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639252633542867908%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Z4MLohw9wIU25IHMP50mTI1nD%2B7RpIVdaLKOR3HPnQ0%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Motivation
> > > > > > > >
> > > > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake,
> > > > > > DuckDB, and
> > > > > > > > Presto/Trino expect to manipulate collections (arrays and maps)
> > > > > inline
> > > > > > > > with a
> > > > > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke
> > UDFs.
> > > > The
> > > > > > > > absence of
> > > > > > > > higher-order collection functions forces verbose query rewrites
> > > > > during
> > > > > > > > migration
> > > > > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1)
> > and
> > > > > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested
> > entry
> > > > > points
> > > > > > > > and,
> > > > > > > > importantly, they can be built on lambda infrastructure that
> > > > already
> > > > > > > > exists in
> > > > > > > > Calcite (CALCITE-3679), so the surface area we add on the Flink
> > > > side
> > > > > is
> > > > > > > > rather
> > > > > > > > small.
> > > > > > > >
> > > > > > > > Summary
> > > > > > > >
> > > > > > > > • Introduce a new logical type FUNCTION that describes the
> > type
> > > > of
> > > > > a
> > > > > > > > lambda
> > > > > > > > (its argument types and its result type). This is the
> > > > type-system
> > > > > > > > foundation
> > > > > > > > every higher-order function needs; it is a
> > > planning/translation
> > > > > > helper
> > > > > > > > type
> > > > > > > > and is not a persisted column type.
> > > > > > > > • Add ARRAY_FILTER(array, element -> predicate), which
> > returns
> > > a
> > > > > new
> > > > > > > > array
> > > > > > > > containing only the elements for which the predicate holds.
> > > > > > > > • Add TRANSFORM(collection, lambda), which applies a lambda
> > to
> > > > > every
> > > > > > > > element of
> > > > > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a
> > > map
> > > > > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new
> > array/map.
> > > > > > > > • Reuse Calcite's lambda parsing, validation and
> > > > > > > > RexLambda/FunctionSqlType
> > > > > > > > infrastructure (CALCITE-3679) rather than inventing a
> > > > > > Flink-specific
> > > > > > > > lambda
> > > > > > > > syntax. The lambda arrow syntax x -> expr and (k, v) ->
> > expr
> > > is
> > > > > > already
> > > > > > > > parseable by the Calcite version Flink bundles (1.41.0).
> > > > > > > >
> > > > > > > > The functions are net-new syntax and are additive: no existing
> > > > query
> > > > > > > > changes
> > > > > > > > behavior. There is no new configuration option — the functions
> > > are
> > > > > > always
> > > > > > > > available once the release ships.
> > > > > > > >
> > > > > > > > Examples:
> > > > > > > >
> > > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2);
> > > > > --
> > > > > > > > [3, 4]
> > > > > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10);
> > > > > --
> > > > > > > > [10, 20, 30]
> > > > > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s ->
> > > CHAR_LENGTH(s));
> > > > > --
> > > > > > > > [1, 2, 3]
> > > > > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100);
> > > > > --
> > > > > > > > {a=100, b=200}
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dominik Bünzli
> > > > > > > > Data, Analytics & AI Engineer
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Sergey
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER)
Posted by Martijn Visser <ma...@apache.org>.
Hi all,
I was wondering if this proposal is still moving forward? I'd love to
see some of these changes land in the upcoming Flink release.
Thanks, Martijn
Op vr 21 aug 2026 om 20:27 schreef David Anderson <da...@apache.org>:
>
> +1 for the overall direction; we've needed this. It will significantly
> improve both usability and readability.
>
> Also +1 for Gustavo's recommendations. The overall scope is just too much
> for one FLIP.
>
> On Fri, Aug 21, 2026 at 8:42 AM Gustavo de Morais <gu...@gmail.com>
> wrote:
>
> > Hi Michal and Dominik,
> >
> > Thanks for working on this - the new document is much more mature than the
> > first one! In general, I think we're in a good direction, +1.
> >
> > A few points:
> >
> > 1) Splitting the FLIP
> >
> > This is a lot of content for careful review, and each function really needs
> > its own discussion. If we include so many in one FLIP, the review will
> > probably take even longer. I'd suggest we focus on the foundation first and
> > get that cleared, and have the function discussions in a second FLIP (and
> > partially on the pull requests).
> >
> > Suggestion:
> > - FLIP 1 (foundation): lambda syntax + FUNCTION type + capture lifting +
> > codegen + Table API/Python etc. It can include a few example functions
> > (e.g. ARRAY_TRANSFORM, ARRAY_FILTER, ARRAY_REDUCE) to exercise the design,
> > but without committing to their shape.
> > - FLIP 2+ (breadth): a second FLIP to discuss the first set of functions,
> > their APIs and shapes.
> >
> > 2) Making it a FLIP document
> >
> > The document currently reads like a long decision log. To make it official
> > as a FLIP, I think we should turn it into a concise, easy-to-read document:
> > - Make it as short as possible with ofc still all the relevant information
> > we need. Right now at 85 pages, it's really only consumable by AI.
> > - Add an index at the top so it's easy to navigate. We often come back to a
> > FLIP many times over the years, so this matters.
> > - Add examples wherever we can - examples are in general super helpful to
> > immediately grasp the content. There are some short ones in the
> > "Composition" section - add more whetever you can.
> > - As orientation, it helps to look at how recent FLIPs are structured. I'm
> > happy to help make sure this lands on our Confluence when we're ready.
> >
> > 3) Content feedback (for later, not the first FLIP)
> >
> > Note: I don't think these belong in the first FLIP we should focus on, so
> > feel free to ignore them for now - just sharing for the future and feel
> > free to not reply directly to these here:
> > - Predicates like exists/forall (Spark) and any_match/all_match/none_match
> > (Trino). Very common, and hard to express well with FILTER + CARDINALITY.
> > - An (element, index) variant of transform/filter, like Spark/Trino
> > transform(a, (x, i) -> ...) with i the 0-based index. Pretty core and
> > useful.
> > - A custom comparator array_sort(array, comparator), as in Spark/Trino
> > (2-arg comparator returning negative/0/positive).
> >
> > 4) EXPLAIN improvement
> >
> > For a capturing lambda, the plan shows the lifted form: the user writes x
> > -> x + base, but the plan renders (x, cap$1) -> (x + cap$1), base. Could we
> > de-lift at render time so EXPLAIN shows intent, not the optimization - i.e.
> > Calc(select=[ARRAY_TRANSFORM(a, x -> (x + base)) AS EXPR$0])? The internal
> > representation can stay as-is.
> >
> > Thanks again for driving this,
> >
> > Kind regards,
> > Gustavo
> >
> > On Sun, 9 Aug 2026 at 10:05, Michal Stutzmann <mi...@gmail.com>
> > wrote:
> >
> > > Hi Gustavo,
> > >
> > > Thanks for looking into this and the pointers. We have now included more
> > > details in the FLIP draft (link shared by Dominik in the previous e-mail:
> > >
> > >
> > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
> > > ).
> > >
> > > In short:
> > > - A UDF called from a lambda body — supported.
> > > - A UDF passed as the lambda, e.g. `ARRAY_TRANSFORM(a, my_udf)`, is not
> > > supported — this would need eta-expansion. The same call written as
> > > `ARRAY_TRANSFORM(a, x -> my_udf(x))` is supported.
> > > - Nesting: supported; arbitrary depth, any combination of built-in and
> > > user-defined, in both directions, with each level capturing the
> > > parameters of all enclosing lambdas plus outer columns.
> > >
> > > All three are covered by a new section in the FLIP draft called Public
> > > Interfaces / 6. Composition, which lists the supported compositions with
> > an
> > > example for each. The mechanism behind it is in Proposed Changes / Lambda
> > > captures.
> > >
> > > Every one of those is covered by an IT case that executes the query and
> > > asserts the result. If you have other combinations in mind, we will add
> > > them.
> > >
> > > Best,
> > > Michal
> > >
> > >
> > > On Fri, Aug 7, 2026 at 4:15 PM Gustavo de Morais <gustavopguto@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Dominik,
> > > >
> > > > Thanks for the reply and for considering these. It's great that you're
> > > both
> > > > working on this. As mentioned, it's a very important feature, but there
> > > are
> > > > indeed some cases that might grow its scope - we should make sure we
> > > design
> > > > it in a complete way that fits the different parts of the project.
> > > >
> > > > I'll try to make some time next week to take a deeper look at what you
> > > > shared. In the meantime, here are a few more pointers you can already
> > > look
> > > > into: you mentioned lambda arguments for user-defined functions - what
> > > > about a UDF as an argument of a lambda function? Nesting is also worth
> > > > considering: multiple nested lambda functions, lambda function -> UDF
> > or
> > > > regular function -> lambda function -> ... etc.
> > > >
> > > > Kind regards,
> > > > Gustavo
> > > >
> > > > On Fri, 7 Aug 2026 at 11:31, <Do...@swisscom.com> wrote:
> > > >
> > > > > Hi Gustavo,
> > > > >
> > > > > Sorry for the late response. I had "a few" discussions with my
> > > colleague
> > > > > Michal Stutzmann (you might remember him from the Flink Forward
> > > > conference
> > > > > in Barcelona) about the FLIP and the features it should cover. Michal
> > > > > extended the proposal<
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
> > > > >
> > > > > and implementation to also support the Table & Python APIs, as well
> > as
> > > > > lambda arguments for user-defined functions. Additionally, we now
> > > support
> > > > > using columns within lambda expressions.
> > > > >
> > > > > We pushed the WIP changes to a branch in our fork, which you can find
> > > > here:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas
> > > > >
> > > > > The set of changes grew larger than expected, but in our opinion they
> > > > > significantly simplify the user experience by reducing the need to
> > work
> > > > > with UDFs or custom code to handle collections.
> > > > >
> > > > > Thanks a lot in advance for taking a look!
> > > > >
> > > > > Best,
> > > > > Dominik
> > > > >
> > > > > From: Gustavo de Morais <gu...@gmail.com>
> > > > > Date: Friday, 31 July 2026 at 12:05
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > > > > (TRANSFORM and ARRAY_FILTER)
> > > > >
> > > > >
> > > > > Be aware: This is an external email.
> > > > >
> > > > >
> > > > >
> > > > > Hey Dominik,
> > > > >
> > > > > Lambda functions are an important building block - not only for these
> > > > > built-in collection functions, but also for UDFs and PTFs whitin the
> > > > whole
> > > > > archictecture. So thanks for the proposal and for wanting to
> > > contribute.
> > > > >
> > > > > I'll take a look at the proposal soon. You mentioned a WIP branch a
> > > > couple
> > > > > of times in the thread - could you share the link so we can take a
> > look
> > > > at
> > > > > that too?
> > > > >
> > > > > Kind regards,
> > > > > Gustavo
> > > > >
> > > > > On Thu, 30 Jul 2026 at 16:39, <Do...@swisscom.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Following up on this FLIP to see if there are any comments or
> > > concerns.
> > > > > >
> > > > > > Since the last mail I added tests for:
> > > > > > - views using ARRAY_FILTER / TRANSFORM
> > > > > > - materialized tables going through expand-and-replan
> > > > > >
> > > > > > Everything is green locally so far.
> > > > > >
> > > > > > Feedback on the planner integration, Calcite handling, missing edge
> > > > > cases,
> > > > > > or the overall direction would be appreciated.
> > > > > >
> > > > > > As I’m still new to the FLIP process, I’d also appreciate some
> > > guidance
> > > > > on
> > > > > > the next steps from here. Many thanks in advance.
> > > > > >
> > > > > > Best,
> > > > > > Dominik
> > > > > >
> > > > > > From: Bünzli Dominik, SCS-INI-DNA-INF <
> > Dominik.Buenzli@swisscom.com>
> > > > > > Date: Friday, 24 July 2026 at 11:17
> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > SQL
> > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > >
> > > > > > Hi Sergey,
> > > > > >
> > > > > > Good catch. I didn’t have these cases in mind. Quickly added test
> > > cases
> > > > > > for them to my branch.
> > > > > >
> > > > > > The view test creates a view like "CREATE VIEW v AS SELECT id,
> > > > > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and
> > > > checks
> > > > > > that the arrays come back right. Same idea for TRANSFORM and the
> > map
> > > > > lambda
> > > > > > (k, v) -> v * 100.
> > > > > >
> > > > > > The materialized table test does the equivalent: a CONTINUOUS table
> > > > using
> > > > > > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster,
> > > > then
> > > > > a
> > > > > > SELECT to confirm the output matches.
> > > > > >
> > > > > > So, both should (to my still limited knowledge), go through the
> > > > > > expand-and-replan path and check real results. If Calcite mangles a
> > > > > lambda,
> > > > > > these tests should fail. Looks good so far, tests are green
> > locally.
> > > > > >
> > > > > > Best,
> > > > > > Dominik
> > > > > >
> > > > > > From: Sergey Nuyanzin <sn...@gmail.com>
> > > > > > Date: Friday, 24 July 2026 at 09:59
> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > SQL
> > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > >
> > > > > >
> > > > > > Be aware: This is an external email.
> > > > > >
> > > > > >
> > > > > >
> > > > > > thanks for the proposal
> > > > > >
> > > > > > and working in this direction
> > > > > >
> > > > > > I haven't checked yet all the things
> > > > > >
> > > > > > however I see at least one missing piece here.
> > > > > >
> > > > > > In FlinkSQL we have views and materialized tables. Both are relying
> > > on
> > > > > > expanded ("rewritten" in Calcite terms) SQL.
> > > > > > And we have already faced a number of issues around that and fixed
> > > > > > them in Calcite (like just a few or them [1], [2], [3], [4], [5],
> > > [6],
> > > > > > [7], [8], [9], [10])
> > > > > >
> > > > > > I'm pretty sure there will be something lambda related. For that
> > > > > > reason I would suggest also having test with views(or materialized
> > > > > > tables, does not matter) based on sql with lambda
> > > > > > and check that SELECT from them still able to produce output
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7660>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7660>>
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7575>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7575>>
> > > > > > [3]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7465>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7465>>
> > > > > > [4]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7480>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7480>>
> > > > > > [5]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7471>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7471>>
> > > > > > [6]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7470>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7470>>
> > > > > > [7]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7466>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7466>>
> > > > > > [8]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7312>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7312>>
> > > > > > [9]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7217>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-7217>>
> > > > > > [10]
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-6944>
> > > > > > <
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0
> > > > > <https://issues.apache.org/jira/browse/CALCITE-6944>>
> > > > > >
> > > > > > On Fri, Jul 24, 2026 at 9:37 AM <Do...@swisscom.com>
> > > wrote:
> > > > > > >
> > > > > > > Hi Gyula
> > > > > > >
> > > > > > > Thanks a lot for your reply and the +1!
> > > > > > >
> > > > > > > On a conceptual level, lambdas and scalar functions / UDFs are
> > > > > distinct.
> > > > > > A lambda isn't a stored/registered value. It's represented by the
> > > > > > planning-only FUNCTION logical type and only exists as an argument
> > to
> > > > > > higher-order functions like ARRAY_FILTER and TRANSFORM.
> > Essentially,
> > > > it's
> > > > > > the wrapper that introduces the element variable x.
> > > > > > >
> > > > > > > Within the lambda you can freely use any scalar function or UDF:
> > > > > > > ARRAY_FILTER(arr, x -> is_positive(x))
> > > > > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100)
> > > > > > >
> > > > > > > What currently doesn't work is passing a bare UDF name in the
> > > lambda
> > > > > > position (i.e. automatic expansion of is_positive into x ->
> > > > > is_positive(x)):
> > > > > > > ARRAY_FILTER(arr, is_positive) -- not supported
> > > > > > >
> > > > > > > We intentionally left this out of scope for now, but it's a
> > > > > > straightforward follow-up if we see demand for it.
> > > > > > >
> > > > > > > Best,
> > > > > > > Dominik
> > > > > > >
> > > > > > > From: Gyula Fóra <gy...@gmail.com>
> > > > > > > Date: Thursday, 23 July 2026 at 16:07
> > > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > > SQL
> > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > > >
> > > > > > >
> > > > > > > Be aware: This is an external email.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hey!
> > > > > > >
> > > > > > > Without a good background knowledge about the syntax in other
> > > > systems,
> > > > > > the
> > > > > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially
> > > with
> > > > > > > lambda functions.
> > > > > > > Overall +1 for this idea from a very high level but it would be
> > > great
> > > > > > with
> > > > > > > someone with more deep context on SQL functions / runtime to
> > chime
> > > > in.
> > > > > > >
> > > > > > > One question I had is how do lambda functions relate to existing
> > > > scalar
> > > > > > > functions / UDFs? Would the user be able to use boolean valued
> > > scalar
> > > > > > > functions where lambdas are shown in the proposal?
> > > > > > >
> > > > > > > Cheers
> > > > > > > Gyula
> > > > > > >
> > > > > > > On Wed, Jul 15, 2026 at 8:40 AM <Do...@swisscom.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I'd like to start a discussion on a FLIP that introduces
> > > > higher-order
> > > > > > > > functions
> > > > > > > > (functions that take a lambda expression as an argument) to
> > Flink
> > > > > SQL,
> > > > > > > > together
> > > > > > > > with the first two built-ins that use them: TRANSFORM and
> > > > > ARRAY_FILTER.
> > > > > > > >
> > > > > > > > FLIP:
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0
> > > > > <
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > > > >
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0
> > > > > <
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > > > >
> > > > > > ><
> > > > > >
> > > > >
> > > >
> > >
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0
> > > > > <
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Motivation
> > > > > > > >
> > > > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake,
> > > > > > DuckDB, and
> > > > > > > > Presto/Trino expect to manipulate collections (arrays and maps)
> > > > > inline
> > > > > > > > with a
> > > > > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke
> > UDFs.
> > > > The
> > > > > > > > absence of
> > > > > > > > higher-order collection functions forces verbose query rewrites
> > > > > during
> > > > > > > > migration
> > > > > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1)
> > and
> > > > > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested
> > entry
> > > > > points
> > > > > > > > and,
> > > > > > > > importantly, they can be built on lambda infrastructure that
> > > > already
> > > > > > > > exists in
> > > > > > > > Calcite (CALCITE-3679), so the surface area we add on the Flink
> > > > side
> > > > > is
> > > > > > > > rather
> > > > > > > > small.
> > > > > > > >
> > > > > > > > Summary
> > > > > > > >
> > > > > > > > • Introduce a new logical type FUNCTION that describes the
> > type
> > > > of
> > > > > a
> > > > > > > > lambda
> > > > > > > > (its argument types and its result type). This is the
> > > > type-system
> > > > > > > > foundation
> > > > > > > > every higher-order function needs; it is a
> > > planning/translation
> > > > > > helper
> > > > > > > > type
> > > > > > > > and is not a persisted column type.
> > > > > > > > • Add ARRAY_FILTER(array, element -> predicate), which
> > returns
> > > a
> > > > > new
> > > > > > > > array
> > > > > > > > containing only the elements for which the predicate holds.
> > > > > > > > • Add TRANSFORM(collection, lambda), which applies a lambda
> > to
> > > > > every
> > > > > > > > element of
> > > > > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a
> > > map
> > > > > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new
> > array/map.
> > > > > > > > • Reuse Calcite's lambda parsing, validation and
> > > > > > > > RexLambda/FunctionSqlType
> > > > > > > > infrastructure (CALCITE-3679) rather than inventing a
> > > > > > Flink-specific
> > > > > > > > lambda
> > > > > > > > syntax. The lambda arrow syntax x -> expr and (k, v) ->
> > expr
> > > is
> > > > > > already
> > > > > > > > parseable by the Calcite version Flink bundles (1.41.0).
> > > > > > > >
> > > > > > > > The functions are net-new syntax and are additive: no existing
> > > > query
> > > > > > > > changes
> > > > > > > > behavior. There is no new configuration option — the functions
> > > are
> > > > > > always
> > > > > > > > available once the release ships.
> > > > > > > >
> > > > > > > > Examples:
> > > > > > > >
> > > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2);
> > > > > --
> > > > > > > > [3, 4]
> > > > > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10);
> > > > > --
> > > > > > > > [10, 20, 30]
> > > > > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s ->
> > > CHAR_LENGTH(s));
> > > > > --
> > > > > > > > [1, 2, 3]
> > > > > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100);
> > > > > --
> > > > > > > > {a=100, b=200}
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dominik Bünzli
> > > > > > > > Data, Analytics & AI Engineer
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Sergey
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER)
Posted by David Anderson <da...@apache.org>.
+1 for the overall direction; we've needed this. It will significantly
improve both usability and readability.
Also +1 for Gustavo's recommendations. The overall scope is just too much
for one FLIP.
On Fri, Aug 21, 2026 at 8:42 AM Gustavo de Morais <gu...@gmail.com>
wrote:
> Hi Michal and Dominik,
>
> Thanks for working on this - the new document is much more mature than the
> first one! In general, I think we're in a good direction, +1.
>
> A few points:
>
> 1) Splitting the FLIP
>
> This is a lot of content for careful review, and each function really needs
> its own discussion. If we include so many in one FLIP, the review will
> probably take even longer. I'd suggest we focus on the foundation first and
> get that cleared, and have the function discussions in a second FLIP (and
> partially on the pull requests).
>
> Suggestion:
> - FLIP 1 (foundation): lambda syntax + FUNCTION type + capture lifting +
> codegen + Table API/Python etc. It can include a few example functions
> (e.g. ARRAY_TRANSFORM, ARRAY_FILTER, ARRAY_REDUCE) to exercise the design,
> but without committing to their shape.
> - FLIP 2+ (breadth): a second FLIP to discuss the first set of functions,
> their APIs and shapes.
>
> 2) Making it a FLIP document
>
> The document currently reads like a long decision log. To make it official
> as a FLIP, I think we should turn it into a concise, easy-to-read document:
> - Make it as short as possible with ofc still all the relevant information
> we need. Right now at 85 pages, it's really only consumable by AI.
> - Add an index at the top so it's easy to navigate. We often come back to a
> FLIP many times over the years, so this matters.
> - Add examples wherever we can - examples are in general super helpful to
> immediately grasp the content. There are some short ones in the
> "Composition" section - add more whetever you can.
> - As orientation, it helps to look at how recent FLIPs are structured. I'm
> happy to help make sure this lands on our Confluence when we're ready.
>
> 3) Content feedback (for later, not the first FLIP)
>
> Note: I don't think these belong in the first FLIP we should focus on, so
> feel free to ignore them for now - just sharing for the future and feel
> free to not reply directly to these here:
> - Predicates like exists/forall (Spark) and any_match/all_match/none_match
> (Trino). Very common, and hard to express well with FILTER + CARDINALITY.
> - An (element, index) variant of transform/filter, like Spark/Trino
> transform(a, (x, i) -> ...) with i the 0-based index. Pretty core and
> useful.
> - A custom comparator array_sort(array, comparator), as in Spark/Trino
> (2-arg comparator returning negative/0/positive).
>
> 4) EXPLAIN improvement
>
> For a capturing lambda, the plan shows the lifted form: the user writes x
> -> x + base, but the plan renders (x, cap$1) -> (x + cap$1), base. Could we
> de-lift at render time so EXPLAIN shows intent, not the optimization - i.e.
> Calc(select=[ARRAY_TRANSFORM(a, x -> (x + base)) AS EXPR$0])? The internal
> representation can stay as-is.
>
> Thanks again for driving this,
>
> Kind regards,
> Gustavo
>
> On Sun, 9 Aug 2026 at 10:05, Michal Stutzmann <mi...@gmail.com>
> wrote:
>
> > Hi Gustavo,
> >
> > Thanks for looking into this and the pointers. We have now included more
> > details in the FLIP draft (link shared by Dominik in the previous e-mail:
> >
> >
> https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
> > ).
> >
> > In short:
> > - A UDF called from a lambda body — supported.
> > - A UDF passed as the lambda, e.g. `ARRAY_TRANSFORM(a, my_udf)`, is not
> > supported — this would need eta-expansion. The same call written as
> > `ARRAY_TRANSFORM(a, x -> my_udf(x))` is supported.
> > - Nesting: supported; arbitrary depth, any combination of built-in and
> > user-defined, in both directions, with each level capturing the
> > parameters of all enclosing lambdas plus outer columns.
> >
> > All three are covered by a new section in the FLIP draft called Public
> > Interfaces / 6. Composition, which lists the supported compositions with
> an
> > example for each. The mechanism behind it is in Proposed Changes / Lambda
> > captures.
> >
> > Every one of those is covered by an IT case that executes the query and
> > asserts the result. If you have other combinations in mind, we will add
> > them.
> >
> > Best,
> > Michal
> >
> >
> > On Fri, Aug 7, 2026 at 4:15 PM Gustavo de Morais <gustavopguto@gmail.com
> >
> > wrote:
> >
> > > Hi Dominik,
> > >
> > > Thanks for the reply and for considering these. It's great that you're
> > both
> > > working on this. As mentioned, it's a very important feature, but there
> > are
> > > indeed some cases that might grow its scope - we should make sure we
> > design
> > > it in a complete way that fits the different parts of the project.
> > >
> > > I'll try to make some time next week to take a deeper look at what you
> > > shared. In the meantime, here are a few more pointers you can already
> > look
> > > into: you mentioned lambda arguments for user-defined functions - what
> > > about a UDF as an argument of a lambda function? Nesting is also worth
> > > considering: multiple nested lambda functions, lambda function -> UDF
> or
> > > regular function -> lambda function -> ... etc.
> > >
> > > Kind regards,
> > > Gustavo
> > >
> > > On Fri, 7 Aug 2026 at 11:31, <Do...@swisscom.com> wrote:
> > >
> > > > Hi Gustavo,
> > > >
> > > > Sorry for the late response. I had "a few" discussions with my
> > colleague
> > > > Michal Stutzmann (you might remember him from the Flink Forward
> > > conference
> > > > in Barcelona) about the FLIP and the features it should cover. Michal
> > > > extended the proposal<
> > > >
> > >
> >
> https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
> > > >
> > > > and implementation to also support the Table & Python APIs, as well
> as
> > > > lambda arguments for user-defined functions. Additionally, we now
> > support
> > > > using columns within lambda expressions.
> > > >
> > > > We pushed the WIP changes to a branch in our fork, which you can find
> > > here:
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas
> > > >
> > > > The set of changes grew larger than expected, but in our opinion they
> > > > significantly simplify the user experience by reducing the need to
> work
> > > > with UDFs or custom code to handle collections.
> > > >
> > > > Thanks a lot in advance for taking a look!
> > > >
> > > > Best,
> > > > Dominik
> > > >
> > > > From: Gustavo de Morais <gu...@gmail.com>
> > > > Date: Friday, 31 July 2026 at 12:05
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > > > (TRANSFORM and ARRAY_FILTER)
> > > >
> > > >
> > > > Be aware: This is an external email.
> > > >
> > > >
> > > >
> > > > Hey Dominik,
> > > >
> > > > Lambda functions are an important building block - not only for these
> > > > built-in collection functions, but also for UDFs and PTFs whitin the
> > > whole
> > > > archictecture. So thanks for the proposal and for wanting to
> > contribute.
> > > >
> > > > I'll take a look at the proposal soon. You mentioned a WIP branch a
> > > couple
> > > > of times in the thread - could you share the link so we can take a
> look
> > > at
> > > > that too?
> > > >
> > > > Kind regards,
> > > > Gustavo
> > > >
> > > > On Thu, 30 Jul 2026 at 16:39, <Do...@swisscom.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Following up on this FLIP to see if there are any comments or
> > concerns.
> > > > >
> > > > > Since the last mail I added tests for:
> > > > > - views using ARRAY_FILTER / TRANSFORM
> > > > > - materialized tables going through expand-and-replan
> > > > >
> > > > > Everything is green locally so far.
> > > > >
> > > > > Feedback on the planner integration, Calcite handling, missing edge
> > > > cases,
> > > > > or the overall direction would be appreciated.
> > > > >
> > > > > As I’m still new to the FLIP process, I’d also appreciate some
> > guidance
> > > > on
> > > > > the next steps from here. Many thanks in advance.
> > > > >
> > > > > Best,
> > > > > Dominik
> > > > >
> > > > > From: Bünzli Dominik, SCS-INI-DNA-INF <
> Dominik.Buenzli@swisscom.com>
> > > > > Date: Friday, 24 July 2026 at 11:17
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> SQL
> > > > > (TRANSFORM and ARRAY_FILTER)
> > > > >
> > > > > Hi Sergey,
> > > > >
> > > > > Good catch. I didn’t have these cases in mind. Quickly added test
> > cases
> > > > > for them to my branch.
> > > > >
> > > > > The view test creates a view like "CREATE VIEW v AS SELECT id,
> > > > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and
> > > checks
> > > > > that the arrays come back right. Same idea for TRANSFORM and the
> map
> > > > lambda
> > > > > (k, v) -> v * 100.
> > > > >
> > > > > The materialized table test does the equivalent: a CONTINUOUS table
> > > using
> > > > > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster,
> > > then
> > > > a
> > > > > SELECT to confirm the output matches.
> > > > >
> > > > > So, both should (to my still limited knowledge), go through the
> > > > > expand-and-replan path and check real results. If Calcite mangles a
> > > > lambda,
> > > > > these tests should fail. Looks good so far, tests are green
> locally.
> > > > >
> > > > > Best,
> > > > > Dominik
> > > > >
> > > > > From: Sergey Nuyanzin <sn...@gmail.com>
> > > > > Date: Friday, 24 July 2026 at 09:59
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> SQL
> > > > > (TRANSFORM and ARRAY_FILTER)
> > > > >
> > > > >
> > > > > Be aware: This is an external email.
> > > > >
> > > > >
> > > > >
> > > > > thanks for the proposal
> > > > >
> > > > > and working in this direction
> > > > >
> > > > > I haven't checked yet all the things
> > > > >
> > > > > however I see at least one missing piece here.
> > > > >
> > > > > In FlinkSQL we have views and materialized tables. Both are relying
> > on
> > > > > expanded ("rewritten" in Calcite terms) SQL.
> > > > > And we have already faced a number of issues around that and fixed
> > > > > them in Calcite (like just a few or them [1], [2], [3], [4], [5],
> > [6],
> > > > > [7], [8], [9], [10])
> > > > >
> > > > > I'm pretty sure there will be something lambda related. For that
> > > > > reason I would suggest also having test with views(or materialized
> > > > > tables, does not matter) based on sql with lambda
> > > > > and check that SELECT from them still able to produce output
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7660>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7660>>
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7575>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7575>>
> > > > > [3]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7465>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7465>>
> > > > > [4]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7480>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7480>>
> > > > > [5]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7471>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7471>>
> > > > > [6]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7470>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7470>>
> > > > > [7]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7466>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7466>>
> > > > > [8]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7312>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7312>>
> > > > > [9]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7217>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-7217>>
> > > > > [10]
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-6944>
> > > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0
> > > > <https://issues.apache.org/jira/browse/CALCITE-6944>>
> > > > >
> > > > > On Fri, Jul 24, 2026 at 9:37 AM <Do...@swisscom.com>
> > wrote:
> > > > > >
> > > > > > Hi Gyula
> > > > > >
> > > > > > Thanks a lot for your reply and the +1!
> > > > > >
> > > > > > On a conceptual level, lambdas and scalar functions / UDFs are
> > > > distinct.
> > > > > A lambda isn't a stored/registered value. It's represented by the
> > > > > planning-only FUNCTION logical type and only exists as an argument
> to
> > > > > higher-order functions like ARRAY_FILTER and TRANSFORM.
> Essentially,
> > > it's
> > > > > the wrapper that introduces the element variable x.
> > > > > >
> > > > > > Within the lambda you can freely use any scalar function or UDF:
> > > > > > ARRAY_FILTER(arr, x -> is_positive(x))
> > > > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100)
> > > > > >
> > > > > > What currently doesn't work is passing a bare UDF name in the
> > lambda
> > > > > position (i.e. automatic expansion of is_positive into x ->
> > > > is_positive(x)):
> > > > > > ARRAY_FILTER(arr, is_positive) -- not supported
> > > > > >
> > > > > > We intentionally left this out of scope for now, but it's a
> > > > > straightforward follow-up if we see demand for it.
> > > > > >
> > > > > > Best,
> > > > > > Dominik
> > > > > >
> > > > > > From: Gyula Fóra <gy...@gmail.com>
> > > > > > Date: Thursday, 23 July 2026 at 16:07
> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > SQL
> > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > >
> > > > > >
> > > > > > Be aware: This is an external email.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hey!
> > > > > >
> > > > > > Without a good background knowledge about the syntax in other
> > > systems,
> > > > > the
> > > > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially
> > with
> > > > > > lambda functions.
> > > > > > Overall +1 for this idea from a very high level but it would be
> > great
> > > > > with
> > > > > > someone with more deep context on SQL functions / runtime to
> chime
> > > in.
> > > > > >
> > > > > > One question I had is how do lambda functions relate to existing
> > > scalar
> > > > > > functions / UDFs? Would the user be able to use boolean valued
> > scalar
> > > > > > functions where lambdas are shown in the proposal?
> > > > > >
> > > > > > Cheers
> > > > > > Gyula
> > > > > >
> > > > > > On Wed, Jul 15, 2026 at 8:40 AM <Do...@swisscom.com>
> > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I'd like to start a discussion on a FLIP that introduces
> > > higher-order
> > > > > > > functions
> > > > > > > (functions that take a lambda expression as an argument) to
> Flink
> > > > SQL,
> > > > > > > together
> > > > > > > with the first two built-ins that use them: TRANSFORM and
> > > > ARRAY_FILTER.
> > > > > > >
> > > > > > > FLIP:
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0
> > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > > >
> > > > > <
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0
> > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > > >
> > > > > ><
> > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0
> > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > > >
> > > > > >
> > > > > > >
> > > > > > > Motivation
> > > > > > >
> > > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake,
> > > > > DuckDB, and
> > > > > > > Presto/Trino expect to manipulate collections (arrays and maps)
> > > > inline
> > > > > > > with a
> > > > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke
> UDFs.
> > > The
> > > > > > > absence of
> > > > > > > higher-order collection functions forces verbose query rewrites
> > > > during
> > > > > > > migration
> > > > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1)
> and
> > > > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested
> entry
> > > > points
> > > > > > > and,
> > > > > > > importantly, they can be built on lambda infrastructure that
> > > already
> > > > > > > exists in
> > > > > > > Calcite (CALCITE-3679), so the surface area we add on the Flink
> > > side
> > > > is
> > > > > > > rather
> > > > > > > small.
> > > > > > >
> > > > > > > Summary
> > > > > > >
> > > > > > > • Introduce a new logical type FUNCTION that describes the
> type
> > > of
> > > > a
> > > > > > > lambda
> > > > > > > (its argument types and its result type). This is the
> > > type-system
> > > > > > > foundation
> > > > > > > every higher-order function needs; it is a
> > planning/translation
> > > > > helper
> > > > > > > type
> > > > > > > and is not a persisted column type.
> > > > > > > • Add ARRAY_FILTER(array, element -> predicate), which
> returns
> > a
> > > > new
> > > > > > > array
> > > > > > > containing only the elements for which the predicate holds.
> > > > > > > • Add TRANSFORM(collection, lambda), which applies a lambda
> to
> > > > every
> > > > > > > element of
> > > > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a
> > map
> > > > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new
> array/map.
> > > > > > > • Reuse Calcite's lambda parsing, validation and
> > > > > > > RexLambda/FunctionSqlType
> > > > > > > infrastructure (CALCITE-3679) rather than inventing a
> > > > > Flink-specific
> > > > > > > lambda
> > > > > > > syntax. The lambda arrow syntax x -> expr and (k, v) ->
> expr
> > is
> > > > > already
> > > > > > > parseable by the Calcite version Flink bundles (1.41.0).
> > > > > > >
> > > > > > > The functions are net-new syntax and are additive: no existing
> > > query
> > > > > > > changes
> > > > > > > behavior. There is no new configuration option — the functions
> > are
> > > > > always
> > > > > > > available once the release ships.
> > > > > > >
> > > > > > > Examples:
> > > > > > >
> > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2);
> > > > --
> > > > > > > [3, 4]
> > > > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10);
> > > > --
> > > > > > > [10, 20, 30]
> > > > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s ->
> > CHAR_LENGTH(s));
> > > > --
> > > > > > > [1, 2, 3]
> > > > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100);
> > > > --
> > > > > > > {a=100, b=200}
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dominik Bünzli
> > > > > > > Data, Analytics & AI Engineer
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Sergey
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER)
Posted by Gustavo de Morais <gu...@gmail.com>.
Hi Michal and Dominik,
Thanks for working on this - the new document is much more mature than the
first one! In general, I think we're in a good direction, +1.
A few points:
1) Splitting the FLIP
This is a lot of content for careful review, and each function really needs
its own discussion. If we include so many in one FLIP, the review will
probably take even longer. I'd suggest we focus on the foundation first and
get that cleared, and have the function discussions in a second FLIP (and
partially on the pull requests).
Suggestion:
- FLIP 1 (foundation): lambda syntax + FUNCTION type + capture lifting +
codegen + Table API/Python etc. It can include a few example functions
(e.g. ARRAY_TRANSFORM, ARRAY_FILTER, ARRAY_REDUCE) to exercise the design,
but without committing to their shape.
- FLIP 2+ (breadth): a second FLIP to discuss the first set of functions,
their APIs and shapes.
2) Making it a FLIP document
The document currently reads like a long decision log. To make it official
as a FLIP, I think we should turn it into a concise, easy-to-read document:
- Make it as short as possible with ofc still all the relevant information
we need. Right now at 85 pages, it's really only consumable by AI.
- Add an index at the top so it's easy to navigate. We often come back to a
FLIP many times over the years, so this matters.
- Add examples wherever we can - examples are in general super helpful to
immediately grasp the content. There are some short ones in the
"Composition" section - add more whetever you can.
- As orientation, it helps to look at how recent FLIPs are structured. I'm
happy to help make sure this lands on our Confluence when we're ready.
3) Content feedback (for later, not the first FLIP)
Note: I don't think these belong in the first FLIP we should focus on, so
feel free to ignore them for now - just sharing for the future and feel
free to not reply directly to these here:
- Predicates like exists/forall (Spark) and any_match/all_match/none_match
(Trino). Very common, and hard to express well with FILTER + CARDINALITY.
- An (element, index) variant of transform/filter, like Spark/Trino
transform(a, (x, i) -> ...) with i the 0-based index. Pretty core and
useful.
- A custom comparator array_sort(array, comparator), as in Spark/Trino
(2-arg comparator returning negative/0/positive).
4) EXPLAIN improvement
For a capturing lambda, the plan shows the lifted form: the user writes x
-> x + base, but the plan renders (x, cap$1) -> (x + cap$1), base. Could we
de-lift at render time so EXPLAIN shows intent, not the optimization - i.e.
Calc(select=[ARRAY_TRANSFORM(a, x -> (x + base)) AS EXPR$0])? The internal
representation can stay as-is.
Thanks again for driving this,
Kind regards,
Gustavo
On Sun, 9 Aug 2026 at 10:05, Michal Stutzmann <mi...@gmail.com>
wrote:
> Hi Gustavo,
>
> Thanks for looking into this and the pointers. We have now included more
> details in the FLIP draft (link shared by Dominik in the previous e-mail:
>
> https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
> ).
>
> In short:
> - A UDF called from a lambda body — supported.
> - A UDF passed as the lambda, e.g. `ARRAY_TRANSFORM(a, my_udf)`, is not
> supported — this would need eta-expansion. The same call written as
> `ARRAY_TRANSFORM(a, x -> my_udf(x))` is supported.
> - Nesting: supported; arbitrary depth, any combination of built-in and
> user-defined, in both directions, with each level capturing the
> parameters of all enclosing lambdas plus outer columns.
>
> All three are covered by a new section in the FLIP draft called Public
> Interfaces / 6. Composition, which lists the supported compositions with an
> example for each. The mechanism behind it is in Proposed Changes / Lambda
> captures.
>
> Every one of those is covered by an IT case that executes the query and
> asserts the result. If you have other combinations in mind, we will add
> them.
>
> Best,
> Michal
>
>
> On Fri, Aug 7, 2026 at 4:15 PM Gustavo de Morais <gu...@gmail.com>
> wrote:
>
> > Hi Dominik,
> >
> > Thanks for the reply and for considering these. It's great that you're
> both
> > working on this. As mentioned, it's a very important feature, but there
> are
> > indeed some cases that might grow its scope - we should make sure we
> design
> > it in a complete way that fits the different parts of the project.
> >
> > I'll try to make some time next week to take a deeper look at what you
> > shared. In the meantime, here are a few more pointers you can already
> look
> > into: you mentioned lambda arguments for user-defined functions - what
> > about a UDF as an argument of a lambda function? Nesting is also worth
> > considering: multiple nested lambda functions, lambda function -> UDF or
> > regular function -> lambda function -> ... etc.
> >
> > Kind regards,
> > Gustavo
> >
> > On Fri, 7 Aug 2026 at 11:31, <Do...@swisscom.com> wrote:
> >
> > > Hi Gustavo,
> > >
> > > Sorry for the late response. I had "a few" discussions with my
> colleague
> > > Michal Stutzmann (you might remember him from the Flink Forward
> > conference
> > > in Barcelona) about the FLIP and the features it should cover. Michal
> > > extended the proposal<
> > >
> >
> https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
> > >
> > > and implementation to also support the Table & Python APIs, as well as
> > > lambda arguments for user-defined functions. Additionally, we now
> support
> > > using columns within lambda expressions.
> > >
> > > We pushed the WIP changes to a branch in our fork, which you can find
> > here:
> > >
> > >
> > >
> >
> https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas
> > >
> > > The set of changes grew larger than expected, but in our opinion they
> > > significantly simplify the user experience by reducing the need to work
> > > with UDFs or custom code to handle collections.
> > >
> > > Thanks a lot in advance for taking a look!
> > >
> > > Best,
> > > Dominik
> > >
> > > From: Gustavo de Morais <gu...@gmail.com>
> > > Date: Friday, 31 July 2026 at 12:05
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > > (TRANSFORM and ARRAY_FILTER)
> > >
> > >
> > > Be aware: This is an external email.
> > >
> > >
> > >
> > > Hey Dominik,
> > >
> > > Lambda functions are an important building block - not only for these
> > > built-in collection functions, but also for UDFs and PTFs whitin the
> > whole
> > > archictecture. So thanks for the proposal and for wanting to
> contribute.
> > >
> > > I'll take a look at the proposal soon. You mentioned a WIP branch a
> > couple
> > > of times in the thread - could you share the link so we can take a look
> > at
> > > that too?
> > >
> > > Kind regards,
> > > Gustavo
> > >
> > > On Thu, 30 Jul 2026 at 16:39, <Do...@swisscom.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Following up on this FLIP to see if there are any comments or
> concerns.
> > > >
> > > > Since the last mail I added tests for:
> > > > - views using ARRAY_FILTER / TRANSFORM
> > > > - materialized tables going through expand-and-replan
> > > >
> > > > Everything is green locally so far.
> > > >
> > > > Feedback on the planner integration, Calcite handling, missing edge
> > > cases,
> > > > or the overall direction would be appreciated.
> > > >
> > > > As I’m still new to the FLIP process, I’d also appreciate some
> guidance
> > > on
> > > > the next steps from here. Many thanks in advance.
> > > >
> > > > Best,
> > > > Dominik
> > > >
> > > > From: Bünzli Dominik, SCS-INI-DNA-INF <Do...@swisscom.com>
> > > > Date: Friday, 24 July 2026 at 11:17
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > > > (TRANSFORM and ARRAY_FILTER)
> > > >
> > > > Hi Sergey,
> > > >
> > > > Good catch. I didn’t have these cases in mind. Quickly added test
> cases
> > > > for them to my branch.
> > > >
> > > > The view test creates a view like "CREATE VIEW v AS SELECT id,
> > > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and
> > checks
> > > > that the arrays come back right. Same idea for TRANSFORM and the map
> > > lambda
> > > > (k, v) -> v * 100.
> > > >
> > > > The materialized table test does the equivalent: a CONTINUOUS table
> > using
> > > > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster,
> > then
> > > a
> > > > SELECT to confirm the output matches.
> > > >
> > > > So, both should (to my still limited knowledge), go through the
> > > > expand-and-replan path and check real results. If Calcite mangles a
> > > lambda,
> > > > these tests should fail. Looks good so far, tests are green locally.
> > > >
> > > > Best,
> > > > Dominik
> > > >
> > > > From: Sergey Nuyanzin <sn...@gmail.com>
> > > > Date: Friday, 24 July 2026 at 09:59
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > > > (TRANSFORM and ARRAY_FILTER)
> > > >
> > > >
> > > > Be aware: This is an external email.
> > > >
> > > >
> > > >
> > > > thanks for the proposal
> > > >
> > > > and working in this direction
> > > >
> > > > I haven't checked yet all the things
> > > >
> > > > however I see at least one missing piece here.
> > > >
> > > > In FlinkSQL we have views and materialized tables. Both are relying
> on
> > > > expanded ("rewritten" in Calcite terms) SQL.
> > > > And we have already faced a number of issues around that and fixed
> > > > them in Calcite (like just a few or them [1], [2], [3], [4], [5],
> [6],
> > > > [7], [8], [9], [10])
> > > >
> > > > I'm pretty sure there will be something lambda related. For that
> > > > reason I would suggest also having test with views(or materialized
> > > > tables, does not matter) based on sql with lambda
> > > > and check that SELECT from them still able to produce output
> > > >
> > > > [1]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7660>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7660>>
> > > > [2]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7575>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7575>>
> > > > [3]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7465>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7465>>
> > > > [4]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7480>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7480>>
> > > > [5]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7471>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7471>>
> > > > [6]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7470>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7470>>
> > > > [7]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7466>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7466>>
> > > > [8]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7312>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7312>>
> > > > [9]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7217>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-7217>>
> > > > [10]
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-6944>
> > > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0
> > > <https://issues.apache.org/jira/browse/CALCITE-6944>>
> > > >
> > > > On Fri, Jul 24, 2026 at 9:37 AM <Do...@swisscom.com>
> wrote:
> > > > >
> > > > > Hi Gyula
> > > > >
> > > > > Thanks a lot for your reply and the +1!
> > > > >
> > > > > On a conceptual level, lambdas and scalar functions / UDFs are
> > > distinct.
> > > > A lambda isn't a stored/registered value. It's represented by the
> > > > planning-only FUNCTION logical type and only exists as an argument to
> > > > higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially,
> > it's
> > > > the wrapper that introduces the element variable x.
> > > > >
> > > > > Within the lambda you can freely use any scalar function or UDF:
> > > > > ARRAY_FILTER(arr, x -> is_positive(x))
> > > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100)
> > > > >
> > > > > What currently doesn't work is passing a bare UDF name in the
> lambda
> > > > position (i.e. automatic expansion of is_positive into x ->
> > > is_positive(x)):
> > > > > ARRAY_FILTER(arr, is_positive) -- not supported
> > > > >
> > > > > We intentionally left this out of scope for now, but it's a
> > > > straightforward follow-up if we see demand for it.
> > > > >
> > > > > Best,
> > > > > Dominik
> > > > >
> > > > > From: Gyula Fóra <gy...@gmail.com>
> > > > > Date: Thursday, 23 July 2026 at 16:07
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> SQL
> > > > (TRANSFORM and ARRAY_FILTER)
> > > > >
> > > > >
> > > > > Be aware: This is an external email.
> > > > >
> > > > >
> > > > >
> > > > > Hey!
> > > > >
> > > > > Without a good background knowledge about the syntax in other
> > systems,
> > > > the
> > > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially
> with
> > > > > lambda functions.
> > > > > Overall +1 for this idea from a very high level but it would be
> great
> > > > with
> > > > > someone with more deep context on SQL functions / runtime to chime
> > in.
> > > > >
> > > > > One question I had is how do lambda functions relate to existing
> > scalar
> > > > > functions / UDFs? Would the user be able to use boolean valued
> scalar
> > > > > functions where lambdas are shown in the proposal?
> > > > >
> > > > > Cheers
> > > > > Gyula
> > > > >
> > > > > On Wed, Jul 15, 2026 at 8:40 AM <Do...@swisscom.com>
> > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I'd like to start a discussion on a FLIP that introduces
> > higher-order
> > > > > > functions
> > > > > > (functions that take a lambda expression as an argument) to Flink
> > > SQL,
> > > > > > together
> > > > > > with the first two built-ins that use them: TRANSFORM and
> > > ARRAY_FILTER.
> > > > > >
> > > > > > FLIP:
> > > > > >
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0
> > > <
> > >
> >
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > >
> > > > <
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0
> > > <
> > >
> >
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > >
> > > > ><
> > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0
> > > <
> > >
> >
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > >
> > > > >
> > > > > >
> > > > > > Motivation
> > > > > >
> > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake,
> > > > DuckDB, and
> > > > > > Presto/Trino expect to manipulate collections (arrays and maps)
> > > inline
> > > > > > with a
> > > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs.
> > The
> > > > > > absence of
> > > > > > higher-order collection functions forces verbose query rewrites
> > > during
> > > > > > migration
> > > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and
> > > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry
> > > points
> > > > > > and,
> > > > > > importantly, they can be built on lambda infrastructure that
> > already
> > > > > > exists in
> > > > > > Calcite (CALCITE-3679), so the surface area we add on the Flink
> > side
> > > is
> > > > > > rather
> > > > > > small.
> > > > > >
> > > > > > Summary
> > > > > >
> > > > > > • Introduce a new logical type FUNCTION that describes the type
> > of
> > > a
> > > > > > lambda
> > > > > > (its argument types and its result type). This is the
> > type-system
> > > > > > foundation
> > > > > > every higher-order function needs; it is a
> planning/translation
> > > > helper
> > > > > > type
> > > > > > and is not a persisted column type.
> > > > > > • Add ARRAY_FILTER(array, element -> predicate), which returns
> a
> > > new
> > > > > > array
> > > > > > containing only the elements for which the predicate holds.
> > > > > > • Add TRANSFORM(collection, lambda), which applies a lambda to
> > > every
> > > > > > element of
> > > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a
> map
> > > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map.
> > > > > > • Reuse Calcite's lambda parsing, validation and
> > > > > > RexLambda/FunctionSqlType
> > > > > > infrastructure (CALCITE-3679) rather than inventing a
> > > > Flink-specific
> > > > > > lambda
> > > > > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr
> is
> > > > already
> > > > > > parseable by the Calcite version Flink bundles (1.41.0).
> > > > > >
> > > > > > The functions are net-new syntax and are additive: no existing
> > query
> > > > > > changes
> > > > > > behavior. There is no new configuration option — the functions
> are
> > > > always
> > > > > > available once the release ships.
> > > > > >
> > > > > > Examples:
> > > > > >
> > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2);
> > > --
> > > > > > [3, 4]
> > > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10);
> > > --
> > > > > > [10, 20, 30]
> > > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s ->
> CHAR_LENGTH(s));
> > > --
> > > > > > [1, 2, 3]
> > > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100);
> > > --
> > > > > > {a=100, b=200}
> > > > > >
> > > > > > Thanks,
> > > > > > Dominik Bünzli
> > > > > > Data, Analytics & AI Engineer
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Sergey
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER)
Posted by Michal Stutzmann <mi...@gmail.com>.
Hi Gustavo,
Thanks for looking into this and the pointers. We have now included more
details in the FLIP draft (link shared by Dominik in the previous e-mail:
https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
).
In short:
- A UDF called from a lambda body — supported.
- A UDF passed as the lambda, e.g. `ARRAY_TRANSFORM(a, my_udf)`, is not
supported — this would need eta-expansion. The same call written as
`ARRAY_TRANSFORM(a, x -> my_udf(x))` is supported.
- Nesting: supported; arbitrary depth, any combination of built-in and
user-defined, in both directions, with each level capturing the
parameters of all enclosing lambdas plus outer columns.
All three are covered by a new section in the FLIP draft called Public
Interfaces / 6. Composition, which lists the supported compositions with an
example for each. The mechanism behind it is in Proposed Changes / Lambda
captures.
Every one of those is covered by an IT case that executes the query and
asserts the result. If you have other combinations in mind, we will add
them.
Best,
Michal
On Fri, Aug 7, 2026 at 4:15 PM Gustavo de Morais <gu...@gmail.com>
wrote:
> Hi Dominik,
>
> Thanks for the reply and for considering these. It's great that you're both
> working on this. As mentioned, it's a very important feature, but there are
> indeed some cases that might grow its scope - we should make sure we design
> it in a complete way that fits the different parts of the project.
>
> I'll try to make some time next week to take a deeper look at what you
> shared. In the meantime, here are a few more pointers you can already look
> into: you mentioned lambda arguments for user-defined functions - what
> about a UDF as an argument of a lambda function? Nesting is also worth
> considering: multiple nested lambda functions, lambda function -> UDF or
> regular function -> lambda function -> ... etc.
>
> Kind regards,
> Gustavo
>
> On Fri, 7 Aug 2026 at 11:31, <Do...@swisscom.com> wrote:
>
> > Hi Gustavo,
> >
> > Sorry for the late response. I had "a few" discussions with my colleague
> > Michal Stutzmann (you might remember him from the Flink Forward
> conference
> > in Barcelona) about the FLIP and the features it should cover. Michal
> > extended the proposal<
> >
> https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
> >
> > and implementation to also support the Table & Python APIs, as well as
> > lambda arguments for user-defined functions. Additionally, we now support
> > using columns within lambda expressions.
> >
> > We pushed the WIP changes to a branch in our fork, which you can find
> here:
> >
> >
> >
> https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas
> >
> > The set of changes grew larger than expected, but in our opinion they
> > significantly simplify the user experience by reducing the need to work
> > with UDFs or custom code to handle collections.
> >
> > Thanks a lot in advance for taking a look!
> >
> > Best,
> > Dominik
> >
> > From: Gustavo de Morais <gu...@gmail.com>
> > Date: Friday, 31 July 2026 at 12:05
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > (TRANSFORM and ARRAY_FILTER)
> >
> >
> > Be aware: This is an external email.
> >
> >
> >
> > Hey Dominik,
> >
> > Lambda functions are an important building block - not only for these
> > built-in collection functions, but also for UDFs and PTFs whitin the
> whole
> > archictecture. So thanks for the proposal and for wanting to contribute.
> >
> > I'll take a look at the proposal soon. You mentioned a WIP branch a
> couple
> > of times in the thread - could you share the link so we can take a look
> at
> > that too?
> >
> > Kind regards,
> > Gustavo
> >
> > On Thu, 30 Jul 2026 at 16:39, <Do...@swisscom.com> wrote:
> >
> > > Hi all,
> > >
> > > Following up on this FLIP to see if there are any comments or concerns.
> > >
> > > Since the last mail I added tests for:
> > > - views using ARRAY_FILTER / TRANSFORM
> > > - materialized tables going through expand-and-replan
> > >
> > > Everything is green locally so far.
> > >
> > > Feedback on the planner integration, Calcite handling, missing edge
> > cases,
> > > or the overall direction would be appreciated.
> > >
> > > As I’m still new to the FLIP process, I’d also appreciate some guidance
> > on
> > > the next steps from here. Many thanks in advance.
> > >
> > > Best,
> > > Dominik
> > >
> > > From: Bünzli Dominik, SCS-INI-DNA-INF <Do...@swisscom.com>
> > > Date: Friday, 24 July 2026 at 11:17
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > > (TRANSFORM and ARRAY_FILTER)
> > >
> > > Hi Sergey,
> > >
> > > Good catch. I didn’t have these cases in mind. Quickly added test cases
> > > for them to my branch.
> > >
> > > The view test creates a view like "CREATE VIEW v AS SELECT id,
> > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and
> checks
> > > that the arrays come back right. Same idea for TRANSFORM and the map
> > lambda
> > > (k, v) -> v * 100.
> > >
> > > The materialized table test does the equivalent: a CONTINUOUS table
> using
> > > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster,
> then
> > a
> > > SELECT to confirm the output matches.
> > >
> > > So, both should (to my still limited knowledge), go through the
> > > expand-and-replan path and check real results. If Calcite mangles a
> > lambda,
> > > these tests should fail. Looks good so far, tests are green locally.
> > >
> > > Best,
> > > Dominik
> > >
> > > From: Sergey Nuyanzin <sn...@gmail.com>
> > > Date: Friday, 24 July 2026 at 09:59
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > > (TRANSFORM and ARRAY_FILTER)
> > >
> > >
> > > Be aware: This is an external email.
> > >
> > >
> > >
> > > thanks for the proposal
> > >
> > > and working in this direction
> > >
> > > I haven't checked yet all the things
> > >
> > > however I see at least one missing piece here.
> > >
> > > In FlinkSQL we have views and materialized tables. Both are relying on
> > > expanded ("rewritten" in Calcite terms) SQL.
> > > And we have already faced a number of issues around that and fixed
> > > them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6],
> > > [7], [8], [9], [10])
> > >
> > > I'm pretty sure there will be something lambda related. For that
> > > reason I would suggest also having test with views(or materialized
> > > tables, does not matter) based on sql with lambda
> > > and check that SELECT from them still able to produce output
> > >
> > > [1]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7660>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7660>>
> > > [2]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7575>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7575>>
> > > [3]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7465>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7465>>
> > > [4]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7480>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7480>>
> > > [5]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7471>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7471>>
> > > [6]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7470>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7470>>
> > > [7]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7466>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7466>>
> > > [8]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7312>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7312>>
> > > [9]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7217>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-7217>>
> > > [10]
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-6944>
> > > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0
> > <https://issues.apache.org/jira/browse/CALCITE-6944>>
> > >
> > > On Fri, Jul 24, 2026 at 9:37 AM <Do...@swisscom.com> wrote:
> > > >
> > > > Hi Gyula
> > > >
> > > > Thanks a lot for your reply and the +1!
> > > >
> > > > On a conceptual level, lambdas and scalar functions / UDFs are
> > distinct.
> > > A lambda isn't a stored/registered value. It's represented by the
> > > planning-only FUNCTION logical type and only exists as an argument to
> > > higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially,
> it's
> > > the wrapper that introduces the element variable x.
> > > >
> > > > Within the lambda you can freely use any scalar function or UDF:
> > > > ARRAY_FILTER(arr, x -> is_positive(x))
> > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100)
> > > >
> > > > What currently doesn't work is passing a bare UDF name in the lambda
> > > position (i.e. automatic expansion of is_positive into x ->
> > is_positive(x)):
> > > > ARRAY_FILTER(arr, is_positive) -- not supported
> > > >
> > > > We intentionally left this out of scope for now, but it's a
> > > straightforward follow-up if we see demand for it.
> > > >
> > > > Best,
> > > > Dominik
> > > >
> > > > From: Gyula Fóra <gy...@gmail.com>
> > > > Date: Thursday, 23 July 2026 at 16:07
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > > (TRANSFORM and ARRAY_FILTER)
> > > >
> > > >
> > > > Be aware: This is an external email.
> > > >
> > > >
> > > >
> > > > Hey!
> > > >
> > > > Without a good background knowledge about the syntax in other
> systems,
> > > the
> > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially with
> > > > lambda functions.
> > > > Overall +1 for this idea from a very high level but it would be great
> > > with
> > > > someone with more deep context on SQL functions / runtime to chime
> in.
> > > >
> > > > One question I had is how do lambda functions relate to existing
> scalar
> > > > functions / UDFs? Would the user be able to use boolean valued scalar
> > > > functions where lambdas are shown in the proposal?
> > > >
> > > > Cheers
> > > > Gyula
> > > >
> > > > On Wed, Jul 15, 2026 at 8:40 AM <Do...@swisscom.com>
> wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to start a discussion on a FLIP that introduces
> higher-order
> > > > > functions
> > > > > (functions that take a lambda expression as an argument) to Flink
> > SQL,
> > > > > together
> > > > > with the first two built-ins that use them: TRANSFORM and
> > ARRAY_FILTER.
> > > > >
> > > > > FLIP:
> > > > >
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0
> > <
> >
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > >
> > > <
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0
> > <
> >
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > >
> > > ><
> > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0
> > <
> >
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > >
> > > >
> > > > >
> > > > > Motivation
> > > > >
> > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake,
> > > DuckDB, and
> > > > > Presto/Trino expect to manipulate collections (arrays and maps)
> > inline
> > > > > with a
> > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs.
> The
> > > > > absence of
> > > > > higher-order collection functions forces verbose query rewrites
> > during
> > > > > migration
> > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and
> > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry
> > points
> > > > > and,
> > > > > importantly, they can be built on lambda infrastructure that
> already
> > > > > exists in
> > > > > Calcite (CALCITE-3679), so the surface area we add on the Flink
> side
> > is
> > > > > rather
> > > > > small.
> > > > >
> > > > > Summary
> > > > >
> > > > > • Introduce a new logical type FUNCTION that describes the type
> of
> > a
> > > > > lambda
> > > > > (its argument types and its result type). This is the
> type-system
> > > > > foundation
> > > > > every higher-order function needs; it is a planning/translation
> > > helper
> > > > > type
> > > > > and is not a persisted column type.
> > > > > • Add ARRAY_FILTER(array, element -> predicate), which returns a
> > new
> > > > > array
> > > > > containing only the elements for which the predicate holds.
> > > > > • Add TRANSFORM(collection, lambda), which applies a lambda to
> > every
> > > > > element of
> > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a map
> > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map.
> > > > > • Reuse Calcite's lambda parsing, validation and
> > > > > RexLambda/FunctionSqlType
> > > > > infrastructure (CALCITE-3679) rather than inventing a
> > > Flink-specific
> > > > > lambda
> > > > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is
> > > already
> > > > > parseable by the Calcite version Flink bundles (1.41.0).
> > > > >
> > > > > The functions are net-new syntax and are additive: no existing
> query
> > > > > changes
> > > > > behavior. There is no new configuration option — the functions are
> > > always
> > > > > available once the release ships.
> > > > >
> > > > > Examples:
> > > > >
> > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2);
> > --
> > > > > [3, 4]
> > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10);
> > --
> > > > > [10, 20, 30]
> > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s));
> > --
> > > > > [1, 2, 3]
> > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100);
> > --
> > > > > {a=100, b=200}
> > > > >
> > > > > Thanks,
> > > > > Dominik Bünzli
> > > > > Data, Analytics & AI Engineer
> > > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Sergey
> > >
> >
>
Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER)
Posted by Gustavo de Morais <gu...@gmail.com>.
Hi Dominik,
Thanks for the reply and for considering these. It's great that you're both
working on this. As mentioned, it's a very important feature, but there are
indeed some cases that might grow its scope - we should make sure we design
it in a complete way that fits the different parts of the project.
I'll try to make some time next week to take a deeper look at what you
shared. In the meantime, here are a few more pointers you can already look
into: you mentioned lambda arguments for user-defined functions - what
about a UDF as an argument of a lambda function? Nesting is also worth
considering: multiple nested lambda functions, lambda function -> UDF or
regular function -> lambda function -> ... etc.
Kind regards,
Gustavo
On Fri, 7 Aug 2026 at 11:31, <Do...@swisscom.com> wrote:
> Hi Gustavo,
>
> Sorry for the late response. I had "a few" discussions with my colleague
> Michal Stutzmann (you might remember him from the Flink Forward conference
> in Barcelona) about the FLIP and the features it should cover. Michal
> extended the proposal<
> https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing>
> and implementation to also support the Table & Python APIs, as well as
> lambda arguments for user-defined functions. Additionally, we now support
> using columns within lambda expressions.
>
> We pushed the WIP changes to a branch in our fork, which you can find here:
>
>
> https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas
>
> The set of changes grew larger than expected, but in our opinion they
> significantly simplify the user experience by reducing the need to work
> with UDFs or custom code to handle collections.
>
> Thanks a lot in advance for taking a look!
>
> Best,
> Dominik
>
> From: Gustavo de Morais <gu...@gmail.com>
> Date: Friday, 31 July 2026 at 12:05
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> (TRANSFORM and ARRAY_FILTER)
>
>
> Be aware: This is an external email.
>
>
>
> Hey Dominik,
>
> Lambda functions are an important building block - not only for these
> built-in collection functions, but also for UDFs and PTFs whitin the whole
> archictecture. So thanks for the proposal and for wanting to contribute.
>
> I'll take a look at the proposal soon. You mentioned a WIP branch a couple
> of times in the thread - could you share the link so we can take a look at
> that too?
>
> Kind regards,
> Gustavo
>
> On Thu, 30 Jul 2026 at 16:39, <Do...@swisscom.com> wrote:
>
> > Hi all,
> >
> > Following up on this FLIP to see if there are any comments or concerns.
> >
> > Since the last mail I added tests for:
> > - views using ARRAY_FILTER / TRANSFORM
> > - materialized tables going through expand-and-replan
> >
> > Everything is green locally so far.
> >
> > Feedback on the planner integration, Calcite handling, missing edge
> cases,
> > or the overall direction would be appreciated.
> >
> > As I’m still new to the FLIP process, I’d also appreciate some guidance
> on
> > the next steps from here. Many thanks in advance.
> >
> > Best,
> > Dominik
> >
> > From: Bünzli Dominik, SCS-INI-DNA-INF <Do...@swisscom.com>
> > Date: Friday, 24 July 2026 at 11:17
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > (TRANSFORM and ARRAY_FILTER)
> >
> > Hi Sergey,
> >
> > Good catch. I didn’t have these cases in mind. Quickly added test cases
> > for them to my branch.
> >
> > The view test creates a view like "CREATE VIEW v AS SELECT id,
> > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and checks
> > that the arrays come back right. Same idea for TRANSFORM and the map
> lambda
> > (k, v) -> v * 100.
> >
> > The materialized table test does the equivalent: a CONTINUOUS table using
> > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, then
> a
> > SELECT to confirm the output matches.
> >
> > So, both should (to my still limited knowledge), go through the
> > expand-and-replan path and check real results. If Calcite mangles a
> lambda,
> > these tests should fail. Looks good so far, tests are green locally.
> >
> > Best,
> > Dominik
> >
> > From: Sergey Nuyanzin <sn...@gmail.com>
> > Date: Friday, 24 July 2026 at 09:59
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > (TRANSFORM and ARRAY_FILTER)
> >
> >
> > Be aware: This is an external email.
> >
> >
> >
> > thanks for the proposal
> >
> > and working in this direction
> >
> > I haven't checked yet all the things
> >
> > however I see at least one missing piece here.
> >
> > In FlinkSQL we have views and materialized tables. Both are relying on
> > expanded ("rewritten" in Calcite terms) SQL.
> > And we have already faced a number of issues around that and fixed
> > them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6],
> > [7], [8], [9], [10])
> >
> > I'm pretty sure there will be something lambda related. For that
> > reason I would suggest also having test with views(or materialized
> > tables, does not matter) based on sql with lambda
> > and check that SELECT from them still able to produce output
> >
> > [1]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7660>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7660>>
> > [2]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7575>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7575>>
> > [3]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7465>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7465>>
> > [4]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7480>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7480>>
> > [5]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7471>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7471>>
> > [6]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7470>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7470>>
> > [7]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7466>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7466>>
> > [8]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7312>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7312>>
> > [9]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7217>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-7217>>
> > [10]
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-6944>
> > <
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0
> <https://issues.apache.org/jira/browse/CALCITE-6944>>
> >
> > On Fri, Jul 24, 2026 at 9:37 AM <Do...@swisscom.com> wrote:
> > >
> > > Hi Gyula
> > >
> > > Thanks a lot for your reply and the +1!
> > >
> > > On a conceptual level, lambdas and scalar functions / UDFs are
> distinct.
> > A lambda isn't a stored/registered value. It's represented by the
> > planning-only FUNCTION logical type and only exists as an argument to
> > higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially, it's
> > the wrapper that introduces the element variable x.
> > >
> > > Within the lambda you can freely use any scalar function or UDF:
> > > ARRAY_FILTER(arr, x -> is_positive(x))
> > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100)
> > >
> > > What currently doesn't work is passing a bare UDF name in the lambda
> > position (i.e. automatic expansion of is_positive into x ->
> is_positive(x)):
> > > ARRAY_FILTER(arr, is_positive) -- not supported
> > >
> > > We intentionally left this out of scope for now, but it's a
> > straightforward follow-up if we see demand for it.
> > >
> > > Best,
> > > Dominik
> > >
> > > From: Gyula Fóra <gy...@gmail.com>
> > > Date: Thursday, 23 July 2026 at 16:07
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> > (TRANSFORM and ARRAY_FILTER)
> > >
> > >
> > > Be aware: This is an external email.
> > >
> > >
> > >
> > > Hey!
> > >
> > > Without a good background knowledge about the syntax in other systems,
> > the
> > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially with
> > > lambda functions.
> > > Overall +1 for this idea from a very high level but it would be great
> > with
> > > someone with more deep context on SQL functions / runtime to chime in.
> > >
> > > One question I had is how do lambda functions relate to existing scalar
> > > functions / UDFs? Would the user be able to use boolean valued scalar
> > > functions where lambdas are shown in the proposal?
> > >
> > > Cheers
> > > Gyula
> > >
> > > On Wed, Jul 15, 2026 at 8:40 AM <Do...@swisscom.com> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'd like to start a discussion on a FLIP that introduces higher-order
> > > > functions
> > > > (functions that take a lambda expression as an argument) to Flink
> SQL,
> > > > together
> > > > with the first two built-ins that use them: TRANSFORM and
> ARRAY_FILTER.
> > > >
> > > > FLIP:
> > > >
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0
> <
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> >
> > <
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0
> <
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> >
> > ><
> >
> https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0
> <
> https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> >
> > >
> > > >
> > > > Motivation
> > > >
> > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake,
> > DuckDB, and
> > > > Presto/Trino expect to manipulate collections (arrays and maps)
> inline
> > > > with a
> > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. The
> > > > absence of
> > > > higher-order collection functions forces verbose query rewrites
> during
> > > > migration
> > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and
> > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry
> points
> > > > and,
> > > > importantly, they can be built on lambda infrastructure that already
> > > > exists in
> > > > Calcite (CALCITE-3679), so the surface area we add on the Flink side
> is
> > > > rather
> > > > small.
> > > >
> > > > Summary
> > > >
> > > > • Introduce a new logical type FUNCTION that describes the type of
> a
> > > > lambda
> > > > (its argument types and its result type). This is the type-system
> > > > foundation
> > > > every higher-order function needs; it is a planning/translation
> > helper
> > > > type
> > > > and is not a persisted column type.
> > > > • Add ARRAY_FILTER(array, element -> predicate), which returns a
> new
> > > > array
> > > > containing only the elements for which the predicate holds.
> > > > • Add TRANSFORM(collection, lambda), which applies a lambda to
> every
> > > > element of
> > > > an array (TRANSFORM(array, x -> expr)) or every entry of a map
> > > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map.
> > > > • Reuse Calcite's lambda parsing, validation and
> > > > RexLambda/FunctionSqlType
> > > > infrastructure (CALCITE-3679) rather than inventing a
> > Flink-specific
> > > > lambda
> > > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is
> > already
> > > > parseable by the Calcite version Flink bundles (1.41.0).
> > > >
> > > > The functions are net-new syntax and are additive: no existing query
> > > > changes
> > > > behavior. There is no new configuration option — the functions are
> > always
> > > > available once the release ships.
> > > >
> > > > Examples:
> > > >
> > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2);
> --
> > > > [3, 4]
> > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10);
> --
> > > > [10, 20, 30]
> > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s));
> --
> > > > [1, 2, 3]
> > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100);
> --
> > > > {a=100, b=200}
> > > >
> > > > Thanks,
> > > > Dominik Bünzli
> > > > Data, Analytics & AI Engineer
> > > >
> >
> >
> >
> > --
> > Best regards,
> > Sergey
> >
>