Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Ajantha Bhat <aj...@gmail.com>]

Update on the progress.

I had a meeting today with Yufei and Yun.zou to discuss the UDF proposal.
We covered several key points, though some are still open for further
discussion:

a) *UDF Versioning*: Do we truly need versioning for UDFs at this stage? We
explored the possibility of simplifying the specification by avoiding view
replication, and potentially introducing versioning support later. UDTFs,
being a superset of views in some ways, may not require versioning
initially.

b) *VarArgs Support*: While some query engines may not support vararg
syntax in CREATE FUNCTION, Iceberg UDFs could represent such arguments as
lists when supported by the engine.

c) *Generics in UDFs*: Since Iceberg currently doesn’t support generic
types (e.g., object), we can only map engine-specific types to Iceberg
types. As a result, generic data types will not be supported in the initial
version.

d) *Python Support*: Incorporating Python as a language for SQL UDFs seems
promising, especially given its potential to resolve interoperability
challenges. Some engines, however, require platform version and package
dependency details to execute Python code—this should be captured in the
specification.

*Next Steps*
I will update the proposal document with two primary UDF use cases:

   -

   Policy exchange between engines
   -

   UDTF as a superset of view functionality

The update will include corresponding syntax examples in both SQL and
Python, and detail how each use case is represented in Iceberg metadata.

We also plan to set up regular syncs (open to more interested participants)
to continue refining and finalizing the UDF specification.
- Ajantha


On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <aj...@gmail.com> wrote:

> Hi everyone,
>
> I've updated the design document[1] based on the previous comments.
> Additionally, I've included the SQL UDF syntax supported by various
> vendors, including Dremio, Snowflake, Databricks, and Trino.
>
> I'm happy to schedule a separate sync if a deeper discussion is needed.
> Let's keep moving forward, especially with the renewed interest from the
> community.
>
> [1]
> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>
> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
>> Hey everyone,
>>
>> During the last catalog community sync, there was significant interest in
>> storing UDFs in Iceberg and adding endpoints for UDF handling in the REST
>> catalog spec.
>>
>> I recently discussed this with Yufei to better understand the new
>> requirement of using UDFs for fine-grained access control policies. This
>> expands the use cases beyond just versioned and interoperable UDFs.
>> Additionally, I learnt that many vendors are interested in this feature.
>>
>> Given the strong community interest and support, I’d like to take
>> ownership of this effort and revive the work. I'll be revisiting the
>> document I proposed long back and will share an updated proposal by next
>> week.
>>
>> Looking forward to storing UDFs in Iceberg!
>> - Ajantha
>>
>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>> <dm...@dremio.com.invalid> wrote:
>>
>>> The UDF spec does not require representations to be SQL. It merely does
>>> not specify (in this revision) how other representations are to be written.
>>>
>>> This seems like an easy extension (adding a new type in the
>>> "Representations" section).
>>>
>>> Cheers,
>>> Dmitri.
>>>
>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue <bl...@databricks.com.invalid>
>>> wrote:
>>>
>>>> Right now, SQL is an explicit requirement of the spec. It leaves a way
>>>> for future versions to add different representations later, but only SQL is
>>>> supported. That was also the feedback to my initial skepticism about how it
>>>> would work to add functions.
>>>>
>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>> <dm...@dremio.com.invalid> wrote:
>>>>
>>>>> I do not think the spec is meant to allow only SQL representations,
>>>>> although it is certainly faviouring SQL in examples... It would be nice to
>>>>> add a non-SQL example, indeed.
>>>>>
>>>>> Cheers,
>>>>> Dmitri.
>>>>>
>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <fo...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Coming from PyIceberg, I have concerns as this proposal focuses on
>>>>>> SQL-based engines, while Python-based systems often work with data frames.
>>>>>> Adding imperative languages like Python would make this proposal more
>>>>>> inclusive.
>>>>>>
>>>>>> Kind regards,
>>>>>> Fokko
>>>>>>
>>>>>>
>>>>>>
>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>> piotr.findeisen@gmail.com>:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Walaa, thanks for asking!
>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>> "Without a common standard, the UDFs are hard to share among
>>>>>>> different engines."
>>>>>>> ("Background and Motivation" section).
>>>>>>> I agree with this statement. I don't fully understand yet how the
>>>>>>> proposed design addresses shareability between the engines though.
>>>>>>> I would use some help to understand this better.
>>>>>>>
>>>>>>> Best
>>>>>>> Piotr
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>
>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>
>>>>>>>> Piotr, what do you mean by making user-created functions shareable
>>>>>>>> between engines? Do you mean UDFs written in imperative code?
>>>>>>>>
>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>> >
>>>>>>>> > Hi,
>>>>>>>> >
>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg UDFs are
>>>>>>>> an interesting idea!
>>>>>>>> > Is there a plan to make the user-created functions sharable
>>>>>>>> between the engines?
>>>>>>>> > If so, how would a CREATE FUNCTION statement look like in e..g
>>>>>>>> Spark or Trino?
>>>>>>>> >
>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>> >
>>>>>>>> > Best
>>>>>>>> > Piotr
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>> >>
>>>>>>>> >> I just looked through the proposal and added comments. I think
>>>>>>>> it would be helpful to also have a design doc that covers the choices from
>>>>>>>> the draft spec. For instance, the choice to enumerate all possible function
>>>>>>>> input struts rather than allowing generics and varargs.
>>>>>>>> >>
>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>> >>
>>>>>>>> >> I think that the choice to enumerate function signatures is
>>>>>>>> limiting. It would be nice to see a discussion of the trade-offs and a
>>>>>>>> rationale for the choice. I think it would also be very helpful to have a
>>>>>>>> few representative use cases for this included in the doc. That way the
>>>>>>>> proposal can demonstrate that it solves those use cases with reasonable
>>>>>>>> trade-offs.
>>>>>>>> >> There are a few instances where this is inconsistent with
>>>>>>>> conventions in other specs. For example, using string IDs rather than an
>>>>>>>> integer.
>>>>>>>> >> This uses a very different model for spec versioning than the
>>>>>>>> Iceberg view and table specs. It requires readers to fail if there are any
>>>>>>>> unknown fields, which prevents the spec from adding things that are fully
>>>>>>>> backward-compatible. Other Iceberg specs only require a version change to
>>>>>>>> introduce forward-incompatible changes and I think that this should do the
>>>>>>>> same to avoid confusion.
>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>> version is associated with a single function signature.
>>>>>>>> >> There is no review of SQL syntax for creating functions across
>>>>>>>> engines, so this doesn’t show that the metadata proposed is sufficient for
>>>>>>>> cross-engine use cases.
>>>>>>>> >> The example for a table-valued function shows a SELECT statement
>>>>>>>> and it isn’t clear how this is distinct from a view
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>> >>>
>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>> >>>
>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>> >>> I will wait for a week and If no more review comments, I will
>>>>>>>> raise a PR for spec addition next week.
>>>>>>>> >>>
>>>>>>>> >>> If anyone else is interested, please have a look at the
>>>>>>>> proposal
>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>> >>>
>>>>>>>> >>> - Ajantha
>>>>>>>> >>>
>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> Hi Ajantha,
>>>>>>>> >>>>
>>>>>>>> >>>> I have left some comments. It is an interesting direction, but
>>>>>>>> there might be some details that need to be fine tuned.
>>>>>>>> >>>>
>>>>>>>> >>>> The doc is here [1] for others who might be interested.
>>>>>>>> Resharing since I do not think it was directly linked in the thread.
>>>>>>>> >>>>
>>>>>>>> >>>> [1]
>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>> >>>>
>>>>>>>> >>>> Thanks,
>>>>>>>> >>>> Walaa.
>>>>>>>> >>>>
>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> Hi, just another reminder since we didn't get any review on
>>>>>>>> the proposal.
>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>> >>>>>
>>>>>>>> >>>>> - Ajantha
>>>>>>>> >>>>>
>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Hi everyone,
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> - Ajantha
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Hi All,
>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions and how we
>>>>>>>> want to implement it.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> - Ajantha
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> Thanks Jack. I actually meant scalar/aggregate/table user
>>>>>>>> defined functions. Here are some examples of what I meant in (2):
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> Probably what you referred to is a variation of (1) where
>>>>>>>> the API is data flow/data pipeline API instead of SQL (e.g., Spark Scala).
>>>>>>>> Yes, that is also possible in the very long run :)
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> Thanks,
>>>>>>>> >>>>>>>> Walaa.
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>> already be a huge step forward.
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> -Jack
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF can be
>>>>>>>> used to build a parameterized view.  So, there's definitely a lot in common
>>>>>>>> between UDFs and views.
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>> >>>>>>>>>>>
>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is perceived
>>>>>>>> as a "UDF". There are 2 flavors:
>>>>>>>> >>>>>>>>>>>
>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user whose
>>>>>>>> definition is a composition of other built-in functions/SQL expressions.
>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>> >>>>>>>>>>>
>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are pretty
>>>>>>>> much from (1) and I think those have more analogy to views due to their SQL
>>>>>>>> nature. Agree (2) is not practical to maintain by Iceberg, but I think
>>>>>>>> Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>> >>>>>>>>>>>
>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>> >>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>
>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the proposal,
>>>>>>>> but I think this would be a very difficult area to tackle across engines,
>>>>>>>> languages, and memory models without having a huge performance penalty.
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec (inspired by
>>>>>>>> the view spec) this week to facilitate further discussions.
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common set of
>>>>>>>> functions across engines, I don't see how that is practical when those
>>>>>>>> engines are implemented so differently. Plugging in code -- and especially
>>>>>>>> custom user-supplied code -- seems inherently specialized to me and should
>>>>>>>> be part of the engines' design.
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel we can
>>>>>>>> say exactly the same thing for Iceberg views, but yet we have Iceberg
>>>>>>>> multi-dialect views implemented. Maybe it sounds like we are trying to draw
>>>>>>>> a line between SQL vs other programming language as "code"? but I think SQL
>>>>>>>> is just another type of code, and we are already talking about compiling
>>>>>>>> all these different code dialects to an intermediate representation (using
>>>>>>>> projects like Coral, Substrait), which will be stored as another type of
>>>>>>>> representation of Iceberg view. I think the same functionality can be used
>>>>>>>> for UDFs if developed.
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good idea,
>>>>>>>> even just a multi-dialect one like view, and that can allow engines to for
>>>>>>>> example parse a view SQL, and when a function referenced cannot be
>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the actual
>>>>>>>> proposal published.
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <
>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable and
>>>>>>>> "non-centralized" as views are. The same performance concerns apply to
>>>>>>>> views as well.
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon which
>>>>>>>> engines can build, so the argument that UDFs aren't practical, because
>>>>>>>> engines are different, is probably only a temporary concern.
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try to tackle
>>>>>>>> the idea to make views portable, which is conceptually not that much
>>>>>>>> different from portable UDFs.
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative touch to the
>>>>>>>> idea of having UDFs in Iceberg, especially not in this early stage.
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good idea to add
>>>>>>>> UDFs tracked by Iceberg catalogs. I think that Iceberg primarily deals with
>>>>>>>> things that are centralized, like tables of data. While it would be great
>>>>>>>> to have a common set of functions across engines, I don't see how that is
>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the proposal,
>>>>>>>> but I think this would be a very difficult area to tackle across engines,
>>>>>>>> languages, and memory models without having a huge performance penalty.
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the community
>>>>>>>> interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for storing
>>>>>>>> the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views in that
>>>>>>>> they are associated with tables, but they can accept arguments and produce
>>>>>>>> return values, or even function as inline expressions.
>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino, Snowflake,
>>>>>>>> Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines. Potentially
>>>>>>>> engines can understand the UDFs written by other engines (with the
>>>>>>>> translate layer).
>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature into
>>>>>>>> Iceberg would be a valuable addition, and we're eager to collaborate with
>>>>>>>> the community to develop a UDF specification.
>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a specification
>>>>>>>> to propose to the community.
>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> --
>>>>>>>> >> Ryan Blue
>>>>>>>> >> Databricks
>>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Databricks
>>>>
>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Péter Váry <pe...@gmail.com>]

Congratulations!
Thanks to Ajantha for getting this started, and to Yufei and everyone else
for making it happen.

Yufei Gu <fl...@gmail.com> ezt írta (időpont: 2026. febr. 6., P, 1:02):

> Hi everyone,
>
> The UDF spec has now been merged 🎉
>
> Huge thanks to everyone involved for the thoughtful discussions, detailed
> reviews, and steady progress over many iterations. This really wouldn’t
> have been possible without the time, care, and collaboration from this
> group.
>
> Since the spec work is now complete, I will cancel the UDF spec sync
> meeting moving forward. If new topics or follow up discussions come up, we
> can always restart ad hoc meetings as needed.
>
> Thanks again, and looking forward to the next steps (e.g., REST spec) and
> reference implementations.
>
> Yufei
>
>
> On Tue, Jan 27, 2026 at 10:22 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> Thanks everyone for joining yesterday's UDF sync. We discussed a few
>> things:
>>
>>    - Secure UDFs are a signal, not a guarantee. Security semantics are
>>    engine specific, and administrators decide which engines are trusted. The
>>    spec should avoid enforcing security behavior and clearly state it does not
>>    provide security guarantees.
>>    - Type definitions use Iceberg type JSON, are IDless, and allow nulls
>>    everywhere. No required fields.
>>    - Variant types are allowed and should be explicitly supported.
>>    - SQL dialects stay coarse grained, for example spark or trino.
>>    Version handling is an implementation detail.
>>    - SQL UDFs first, Python UDFs later.
>>    - The spec is close to final, with mostly wording cleanup remaining.
>>
>> The meeting recording is available here: https://youtu.be/HpFry5PlV1g
>>
>> Please take another look at PR
>> https://github.com/apache/iceberg/pull/14117, and vote in the dev
>> mailing thread.
>>
>> Yufei
>>
>>
>> On Tue, Jan 13, 2026 at 12:14 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Thanks everyone for joining yesterday's UDF sync. We focused on next
>>> steps for the global parameter name list and on how to approach default
>>> values:
>>> 1. We made good progress on parameter names by pushing down the
>>> parameter name list to individual overloads and applying constraints on
>>> call conventions.
>>> 2. For default values, the current direction is to let engines decide
>>> how they support them, rather than specifying default value semantics in
>>> the UDF spec at this stage.
>>>
>>> The meeting recording is available here: https://youtu.be/2uKxYr1yX5w
>>>
>>> You can find detailed and corresponding changes in PR
>>> https://github.com/apache/iceberg/pull/14117. The PR is ready for
>>> another look. Feedback is very welcome.
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Jan 12, 2026 at 9:01 AM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> The UDF spec is overall a good sharp. Thanks for all the feedback. In
>>>> today's sync, we could discuss the global parameter name list(drafted one
>>>> google doc for its pros and cons,
>>>> https://docs.google.com/document/d/1ipeEDTlgnBmoH_DJA3HWs6ZiLRaZGSDR-KwS__gc4-M/edit?tab=t.0)
>>>> and whether we need to support the default value at this phase.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Dec 15, 2025 at 2:18 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Thanks everyone for joining today’s UDF sync. We covered several
>>>>> topics:
>>>>>
>>>>> 1. We generally agreed to prefer a simpler form for complex function
>>>>> type signatures, instead of the Iceberg type JSON format (
>>>>> https://iceberg.apache.org/spec/#appendix-c-json-serialization).
>>>>> 2. We discussed the motivations and concerns around having a global
>>>>> parameter name list. Given the trade-offs, we decided to take a deeper look
>>>>> offline.
>>>>> 3. We talked about default values in terms of overload disambiguation
>>>>> and cross-engine support, and agreed that this needs further investigation.
>>>>>
>>>>> Going forward, we’ll use the following Google Doc to record key design
>>>>> decisions and rationale, so we have a shared reference as discussions
>>>>> evolve:
>>>>>
>>>>> https://docs.google.com/document/d/1GKtqFt9gnlJxwo5kfExupDf1jASlgcnDeZv8vEaJQUs/edit?usp=sharing
>>>>>
>>>>> The meeting recording is available here: https://youtu.be/tF7Ur88snR0
>>>>>
>>>>> Thanks Kevin for the proposal. Given the holiday season, we'd like to
>>>>> propose cancelling the next Iceberg UDF Sync scheduled for December 29,
>>>>> 2025. Please let me know if there are any objections.
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Fri, Dec 5, 2025 at 8:51 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> Thanks for joining Monday’s UDF sync!
>>>>>> We covered two updates:
>>>>>>
>>>>>>    - Parameter names moved to the top level, this keeps
>>>>>>    named-argument invocation consistent across overloads.
>>>>>>    - Default value support added, callers can now omit parameters
>>>>>>    that have defaults.
>>>>>>
>>>>>> The recording is available here for more details:
>>>>>> https://youtu.be/6CBMwwbISOE
>>>>>>
>>>>>> I’ve also added more examples demonstrating default values. Please
>>>>>> take a look at the PR: https://github.com/apache/iceberg/pull/14117
>>>>>>
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 26, 2025 at 5:57 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Made a few changes to the spec. The PR(
>>>>>>> https://github.com/apache/iceberg/pull/14117) is ready for another
>>>>>>> look.
>>>>>>> 1. Moved the parameter name list to the top level, so that
>>>>>>> named-argument invocation is consistent across overloads.
>>>>>>> 2. Added support for default value, which allows caller to ignore
>>>>>>> parameters with default values.
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 17, 2025 at 6:42 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap
>>>>>>>> of what we discussed and agreed on:
>>>>>>>>
>>>>>>>>    1. Huaxin walked us through the Spark POC(
>>>>>>>>    https://github.com/apache/iceberg/pull/14505). Thanks Huaxin!
>>>>>>>>    2. When default values are enabled, ambiguous overloads must
>>>>>>>>    not be allowed. We agreed that definition-id alone is insufficient for
>>>>>>>>    conflict detection; additional signature-level check across overloads is
>>>>>>>>    needed.
>>>>>>>>    3. We will explore the idea that parameter names are globally
>>>>>>>>    consistent across multiple overloads. For example, if the overload foo(int
>>>>>>>>    x, int y) exists, foo(int a) is not allowed, but foo(int x) is OK. We could
>>>>>>>>    put a list of names and their docs at the top level. This benefits UX and
>>>>>>>>    may potentially benefit the strategy to pick up the overload to use given
>>>>>>>>    the inputs. We may bring back the parameter names at the representation
>>>>>>>>    level, so that it's more flexible.
>>>>>>>>    4. Should an engine ignore the default values or fail fast if
>>>>>>>>    it doesn’t support it? We concluded that this is up to the engines, and
>>>>>>>>    engines could enforce all parameters to be provided when default values
>>>>>>>>    aren't supported.
>>>>>>>>    5. How do we give the default value for complex types(e.g.
>>>>>>>>    struct)? We should follow the way table spec handles the default value(
>>>>>>>>    https://iceberg.apache.org/spec/#edge-interpolation-algorithm).
>>>>>>>>
>>>>>>>> You can watch the recording here for more details:
>>>>>>>> https://youtu.be/Wxz1nSP48k0
>>>>>>>>
>>>>>>>> Please take another look at PR
>>>>>>>> https://github.com/apache/iceberg/pull/14117 when you get a chance.
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Nov 14, 2025 at 4:29 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> The PR (https://github.com/apache/iceberg/pull/14117) has been
>>>>>>>>> updated, please take another look.
>>>>>>>>> For next Monday’s UDF sync, Huaxin will demo a Spark POC showing
>>>>>>>>> how the current (WIP) UDF spec works. We can also walk through how default
>>>>>>>>> values should be introduced into the spec.
>>>>>>>>>
>>>>>>>>> *TL;DR:* I haven’t added default-value support in the PR yet.
>>>>>>>>> It’s turning out to be trickier than expected because default values make
>>>>>>>>> parameters optional. That introduces ambiguity for overloads like:
>>>>>>>>>
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    foo(int)
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    foo(int, int default 0)
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    foo(int, float default 1.0)
>>>>>>>>>
>>>>>>>>> All of these could be invoked as foo(int). The first question is:
>>>>>>>>> should we allow such overloads to coexist? Probably not, both Postgres and
>>>>>>>>> Snowflake support default values, but neither allows this form of ambiguous
>>>>>>>>> overloading.
>>>>>>>>> Assuming we disallow it, the current UDF definition-id (type
>>>>>>>>> tuple) isn’t sufficient to detect conflicts. We also need to check the
>>>>>>>>> signatures of existing overloads to determine whether a new one is valid.
>>>>>>>>> For example, if (int, int) exists, then neither (int) nor (int, float)
>>>>>>>>> should be allowed.
>>>>>>>>>
>>>>>>>>> Reference:
>>>>>>>>> Postgres behavior on default parameters:
>>>>>>>>> https://www.postgresql.org/docs/current/static/typeconv-func.html
>>>>>>>>> Functions with default arguments match calls that omit zero or
>>>>>>>>> more defaultable parameters. If multiple functions match, the earliest in
>>>>>>>>> the search path is chosen; if they are equally specific, Postgres throws an
>>>>>>>>> “ambiguous function call” error.
>>>>>>>>> Thanks,
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> Thanks for joining today’s community UDF sync! Here’s a quick
>>>>>>>>>> recap of what we discussed and agreed on:
>>>>>>>>>> 1. We’ll keep parameters only at the definition layer to support
>>>>>>>>>> named-argument invocation without extra work from the engine side.
>>>>>>>>>> 2. We agreed to add support for default parameter values.
>>>>>>>>>> 3. Moving forward with a flag for nullable vs. non-nullable
>>>>>>>>>> return values, treating it as a hint rather than a strict rule.
>>>>>>>>>> 4. The udf metadata will preserve the original case of parameter
>>>>>>>>>> names, and engines can decide whether to normalize or convert them.
>>>>>>>>>>
>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>> https://www.youtube.com/watch?v=0i6nKfDNdGw
>>>>>>>>>> Sorry that today's sync wasn't in the Iceberg Dev Event
>>>>>>>>>> calendar(It was removed somehow). I have added it back to Iceberg Dev Event
>>>>>>>>>> as a biweekly sync. Let me know if you have any questions.
>>>>>>>>>>
>>>>>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>>>> for item 1 and 2 soon. The overall content will remain largely the same.
>>>>>>>>>> Please take a look when you get a chance!
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ryan and me are in this meeting, please join
>>>>>>>>>>>
>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>> Monday, November 3 · 9:00 – 10:00am
>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks a lot for the review! Made a few changes to the spec:
>>>>>>>>>>>> 1. Renamed "overload" to "definition"
>>>>>>>>>>>> 2. Changed definition id from long to string with the format of
>>>>>>>>>>>> parameter type tuple.
>>>>>>>>>>>> 3. Explicitly claimed that named argument invocation is
>>>>>>>>>>>> supported
>>>>>>>>>>>> 4. Added a nullable flag for return values
>>>>>>>>>>>> 5. Added a few rules for inputs and returns
>>>>>>>>>>>>
>>>>>>>>>>>> Please find more details in the PR(
>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117). Here is a doc
>>>>>>>>>>>> comparing engine behaviors in terms of named argument invocation, nullable
>>>>>>>>>>>> return value, default values for inputs, null handling:
>>>>>>>>>>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>>>>>>>>>>> This can help for decision making.
>>>>>>>>>>>>
>>>>>>>>>>>> We could discuss the following items in Monday's meeting:
>>>>>>>>>>>> 1. Whether we need to push down parameters to representation
>>>>>>>>>>>> given we want to support named argument invocation, it requires extra
>>>>>>>>>>>> mapping on the engine side.
>>>>>>>>>>>> 2. Whether we support default values.  Spark plans to support
>>>>>>>>>>>> it, Snowflake supports it already.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for joining today’s community UDF sync. Here’s a
>>>>>>>>>>>>> summary of what we discussed and agreed upon:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1. We agreed to use Iceberg types for each field in the
>>>>>>>>>>>>>    UDF spec. This approach ensures consistency with the table spec and allows
>>>>>>>>>>>>>    for straightforward conversion to JSON types.
>>>>>>>>>>>>>    2. Using numeric overload IDs can have side effects, for
>>>>>>>>>>>>>    instance, deleting an overload and later creating a new one with the same
>>>>>>>>>>>>>    signature will use a different numeric id, which points to the same
>>>>>>>>>>>>>    overload. The consensus is to represent the overload ID as a string derived
>>>>>>>>>>>>>    from parameter types and their order.
>>>>>>>>>>>>>    3. Parameter names must remain consistent across all
>>>>>>>>>>>>>    overload versions. We agreed to add explicit rules:
>>>>>>>>>>>>>       1. Parameter names must match across all overload
>>>>>>>>>>>>>       versions.
>>>>>>>>>>>>>       2. Renaming a parameter requires removing all previous
>>>>>>>>>>>>>       versions that use the old name.
>>>>>>>>>>>>>    4. We discussed whether parameters or return types can be
>>>>>>>>>>>>>    nullable and how engines should handle null values. We’ll research SQL
>>>>>>>>>>>>>    standards and common engine behaviors before finalizing. The current
>>>>>>>>>>>>>    leaning is:
>>>>>>>>>>>>>       1. Allow all parameters to be nullable.
>>>>>>>>>>>>>       2. Allow the return type to include an explicit
>>>>>>>>>>>>>       nullability annotation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>>>>> https://youtu.be/IUd-SN0CQRs
>>>>>>>>>>>>> Will update the PR(
>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) soon.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks a lot for joining Monday's UDF sync. Here is the
>>>>>>>>>>>>>> summary:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    1. Replace “definitions version” with “definitions logs”,
>>>>>>>>>>>>>>    which record overload changes and timestamps for informational purposes
>>>>>>>>>>>>>>    only.
>>>>>>>>>>>>>>    2. Use a globally increasing numeric ID for overloads
>>>>>>>>>>>>>>    instead of UUIDs.
>>>>>>>>>>>>>>    3. Add explicit types to all fields in the spec to
>>>>>>>>>>>>>>    clarify the metadata structure.
>>>>>>>>>>>>>>    4. Add null-handling optimization hints to help engines
>>>>>>>>>>>>>>    skip evaluation when inputs are null. (Thanks, Talat!)
>>>>>>>>>>>>>>    5. Function parameters remain ordered, but we don’t need
>>>>>>>>>>>>>>    to introduce IDs for each parameter. We’ll double-check the Java
>>>>>>>>>>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>>>>>>>>>>    nested structs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>>>>>> https://youtu.be/gJnueQODWvs
>>>>>>>>>>>>>> I have updated the PR (
>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) for the above
>>>>>>>>>>>>>> feedback. Please take a look and let me know if you have any questions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <
>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks a lot for joining today's UDF sync. Here is the
>>>>>>>>>>>>>>> summary:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    1. Instead of relying on dynamic inference, the return
>>>>>>>>>>>>>>>    table’s schema for user-defined table functions (UDTFs) should be
>>>>>>>>>>>>>>>    explicitly defined.
>>>>>>>>>>>>>>>    2. Whether a function is a UDTF should be captured as a
>>>>>>>>>>>>>>>    dedicated attribute, rather than being inferred indirectly from the return
>>>>>>>>>>>>>>>    type.
>>>>>>>>>>>>>>>    3. The interpretation of a UDF body (whether it is
>>>>>>>>>>>>>>>    treated as a partial SQL expression or as a full SELECT statement) should
>>>>>>>>>>>>>>>    be determined by engines. Example: `SELECT x +1` vs. `x + 1`. Different
>>>>>>>>>>>>>>>    engines have different takes.
>>>>>>>>>>>>>>>    4. User-defined aggregation functions (UDAFs) are out of
>>>>>>>>>>>>>>>    scope for now.
>>>>>>>>>>>>>>>    5. Each overload should include its own current-version
>>>>>>>>>>>>>>>    field. This avoids relying solely on the global `definition-versions` when
>>>>>>>>>>>>>>>    querying the current version of one overload.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>>>>>>>>>>> I will update the PR(
>>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) shortly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Really appreciated feedback from you all over the past few
>>>>>>>>>>>>>>>> months. I've filed the initial PR for the UDF spec:
>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures
>>>>>>>>>>>>>>>> the consensus we've built and addresses the write amplification concern
>>>>>>>>>>>>>>>> raised in our last discussion.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please take a look and share your thoughts. Happy to
>>>>>>>>>>>>>>>> discuss it further during Monday's meeting as well.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We covered the UDF metadata structure, captured in this
>>>>>>>>>>>>>>>>> doc:
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We also discussed a way to avoid copying every overload
>>>>>>>>>>>>>>>>> into the new metadata JSON when creating a new version. One of ideas is to
>>>>>>>>>>>>>>>>> introduce a global version array, this is not yet reflected in the doc, but
>>>>>>>>>>>>>>>>> I’ll update it shortly. Other key points:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - The latest UDF version will typically be used in
>>>>>>>>>>>>>>>>>    most scenarios, but engines retain the flexibility to choose which version
>>>>>>>>>>>>>>>>>    to execute.
>>>>>>>>>>>>>>>>>    - Keeping the version while referring to an UDF
>>>>>>>>>>>>>>>>>    probably isn't a good idea. Users are responsible for updating downstream
>>>>>>>>>>>>>>>>>    views if they reference older UDF versions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <
>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In
>>>>>>>>>>>>>>>>>> general, we discussed the UDF metadata structure, captured at this doc(
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    1. Each UDF overload has its own return type. e.g.,
>>>>>>>>>>>>>>>>>>    `add(int, int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>>>>>>>>>>    2. Return type should be explicitly specified, no
>>>>>>>>>>>>>>>>>>    implicit or statement-based return type inference should be allowed.
>>>>>>>>>>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>>>>>>>>>>    properties at the overload level.
>>>>>>>>>>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>>>>>>>>>>    5. Introducing a dedicated signature definitions
>>>>>>>>>>>>>>>>>>    section to centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>>>>>>>>>>    6. Whether to have versioned open properties or not.
>>>>>>>>>>>>>>>>>>    Versioned properties can lead to unnecessary copying of a bag of properties
>>>>>>>>>>>>>>>>>>    into each version, while it provides a clear history of properties for any
>>>>>>>>>>>>>>>>>>    future debugging and understanding of the UDF behavior at a
>>>>>>>>>>>>>>>>>>    specific point in time.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <
>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi everyone, here’s the summary from our last sync on
>>>>>>>>>>>>>>>>>>> 8/11. Apologies for the delay!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>>>>>>>>>>       - We agreed to combine overloads with the same
>>>>>>>>>>>>>>>>>>>       name into a single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>>>>>>>>>>       - Listing UDFs will return a list of UDF names,
>>>>>>>>>>>>>>>>>>>       not a list of individual signatures.
>>>>>>>>>>>>>>>>>>>       - Loading a UDF by name will return all of its
>>>>>>>>>>>>>>>>>>>       overloads.
>>>>>>>>>>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>>>>>>>>>>       - A global version number will track changes
>>>>>>>>>>>>>>>>>>>       across the entire UDF entity, it increments monolithically.
>>>>>>>>>>>>>>>>>>>       - Each overload will also maintain its own
>>>>>>>>>>>>>>>>>>>       version (e.g., updated_at_version) to trace changes specific to that
>>>>>>>>>>>>>>>>>>>       overload.
>>>>>>>>>>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>>>>>>>>>>    this stage.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs
>>>>>>>>>>>>>>>>>>>> with multiple signatures under the same name, which can serve both
>>>>>>>>>>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Per my investigation[1], most engines support
>>>>>>>>>>>>>>>>>>>> overloading by arguments and allow implicit conversions like numeric
>>>>>>>>>>>>>>>>>>>> widening (e.g., INT → BIGINT/FLOAT). The resolution approach causes issues
>>>>>>>>>>>>>>>>>>>> like silent behavior change. Here is an example:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs
>>>>>>>>>>>>>>>>>>>>    expected code.
>>>>>>>>>>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>>>>>>>>>>    - Engine’s best-match resolution now binds the same
>>>>>>>>>>>>>>>>>>>>    call to the new overload, changing behavior without modifying the query.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To mitigate this issue, we have to choose between these
>>>>>>>>>>>>>>>>>>>> two access control models:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all
>>>>>>>>>>>>>>>>>>>>    overloads of a function name.
>>>>>>>>>>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to
>>>>>>>>>>>>>>>>>>>>    specific signatures.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The general recommendation is to adopt *Model A.* It
>>>>>>>>>>>>>>>>>>>> trades some precision for safety and simplicity, while eliminating the
>>>>>>>>>>>>>>>>>>>> silent behavior change problem. More details are in this doc[1].
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>> We have discussed how to define function identifiers
>>>>>>>>>>>>>>>>>>>>> (should also handle function overloading). Ryan suggested that we should
>>>>>>>>>>>>>>>>>>>>> check how Spark does it. We can refer to functions using an identifier and
>>>>>>>>>>>>>>>>>>>>> then bind the different signatures to it. So that access policies can be
>>>>>>>>>>>>>>>>>>>>> applied per identifier. This is also linked to how we want to version the
>>>>>>>>>>>>>>>>>>>>> functions when overloading is supported.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I will check more about this and update the proposal
>>>>>>>>>>>>>>>>>>>>> doc.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Please check/subscribe to the dev events calendar for
>>>>>>>>>>>>>>>>>>>>> the next meeting link (Aug 11).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg
>>>>>>>>>>>>>>>>>>>>>> Dev Events" calendar for tomorrow 7/28 at 9AM PT. I
>>>>>>>>>>>>>>>>>>>>>> missed the last one, but i'll be at this one.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> No one joined the sync today. I came to know that
>>>>>>>>>>>>>>>>>>>>>>> Yufei is on holiday, and Ryan and others couldn't make it, similar to the
>>>>>>>>>>>>>>>>>>>>>>> last sync. It seems Yufei might have forgotten to transfer meeting
>>>>>>>>>>>>>>>>>>>>>>> ownership as well, as new members needed admin approval and couldn't join
>>>>>>>>>>>>>>>>>>>>>>> automatically this week. Also, I can understand it is summer holiday season
>>>>>>>>>>>>>>>>>>>>>>> for many.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I've updated the function signature schema and other
>>>>>>>>>>>>>>>>>>>>>>> open points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a
>>>>>>>>>>>>>>>>>>>>>>>> top level field to a field inside “version” along with a representation, so
>>>>>>>>>>>>>>>>>>>>>>>> that properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> That said, it's still an open question whether we
>>>>>>>>>>>>>>>>>>>>>>>> need any non-versioned properties. We can introduce them later if a use
>>>>>>>>>>>>>>>>>>>>>>>> case arises.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate
>>>>>>>>>>>>>>>>>>>>>>>>> from the representation history. Here are reasons:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    1. Each version still enforces a single
>>>>>>>>>>>>>>>>>>>>>>>>>    signature. Although the signatures array is global to the UDF, each version
>>>>>>>>>>>>>>>>>>>>>>>>>    references just one signature ID. Rollbacks to historical versions remain
>>>>>>>>>>>>>>>>>>>>>>>>>    safe.
>>>>>>>>>>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently
>>>>>>>>>>>>>>>>>>>>>>>>>    changing component (signatures) from the more dynamic one (representations)
>>>>>>>>>>>>>>>>>>>>>>>>>    to reduce metadata file size.
>>>>>>>>>>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types,
>>>>>>>>>>>>>>>>>>>>>>>>>    they should remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>> We have discussed the action items from the last
>>>>>>>>>>>>>>>>>>>>>>>>>> sync (*see Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of
>>>>>>>>>>>>>>>>>>>>>>>>>>    the engines and in the roadmaps of many engines. Iceberg will support it.
>>>>>>>>>>>>>>>>>>>>>>>>>>    We will maintain the `FunctionIdentifier` (extends `TableIdentifer` but
>>>>>>>>>>>>>>>>>>>>>>>>>>    also have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in
>>>>>>>>>>>>>>>>>>>>>>>>>>    a bag, we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural
>>>>>>>>>>>>>>>>>>>>>>>>>>    bodies are supported by some engines. Iceberg will support it. Store the
>>>>>>>>>>>>>>>>>>>>>>>>>>    body as it is while creating function by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    - Standardizing the property names
>>>>>>>>>>>>>>>>>>>>>>>>>>    (deterministic, secure).
>>>>>>>>>>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level
>>>>>>>>>>>>>>>>>>>>>>>>>>    replace is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>>>>>>>>>>    - Signature should be associated with
>>>>>>>>>>>>>>>>>>>>>>>>>>    representation?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please
>>>>>>>>>>>>>>>>>>>>>>>>>>    review the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the
>>>>>>>>>>>>>>>>>>>>>>>>>>> whole metadata is encrypted, we don't have to worry about just hiding the
>>>>>>>>>>>>>>>>>>>>>>>>>>> UDF body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling
>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown are required.We will need a named key(e.g., secure) somewhere, no
>>>>>>>>>>>>>>>>>>>>>>>>>>>> matter if it is a top level property or a key as a part of the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>> properties. So that both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec would require? Would it require hiding the UDF definition from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users and require specific pushdown cases be disabled? The use cases seem
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> valid, but I'm trying to understand the requirements this places on engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and why it needs to be part of the spec, rather than part of the properties
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the UDF.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    concealing the UDF body and details like the list of imports, some of them
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    aren’t applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    runs in an isolated environment. Again, this typically doesn’t apply to SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Time: For example, secure UDFs may disable certain optimizations—such as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    predicate pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point that the secure flag is primarily an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instruction to the engine to behave differently. While it's largely an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine-side behavior, we still need to include this flag in the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition to indicate whether a UDF is secure, especially considering the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> perf penalty introduced by scenario #3. We should clearly recommend that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users avoid marking UDFs as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supporting a "secure" UDF? What use case are you addressing and what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specifically changes about how the UDF is handled? If the idea is to hide
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDF definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine". When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but whether those statements run within a single transaction should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> treated as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “secure UDF” option supported by some engines[1], so the body and any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sensitive details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    syntax supported by different engines (Snowflake, databricks, Dremio,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Trino, OSS spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    separator, like $$ or '' or none. Action item was to check whether engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    support multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    overloading. Need to check whether these engines support function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    overloading for SQL UDFs. Postgres supports it! If yes, need to adopt the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    discussed the deterministic flag and concluded that we keep the independent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    fields (like deterministic) in spec only if the majority of engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    supports it. Else it will be passed in a property bag (engine specific).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    And it is the engine's responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> document here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    support; the majority agreed *not to*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    (see recording for details).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning —
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    it will be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    each catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    existing table/view spec — e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    location will remain *required* for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    document
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    vote once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> community sync for the UDF project. Everyone’s welcome to drop in and share
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ideas! Here is the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun.zou to discuss the UDF proposal. We covered several key points, though
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some are still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines may not support vararg syntax in CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION, Iceberg UDFs could represent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently doesn’t support generic types (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> object), we can only map
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Python as a language for SQL UDFs seems promising, especially given its
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> potential to resolve interoperability challenges. Some engines, however,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> require platform version and package dependency details to execute Python
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code—this should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with two primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (open to more interested participants) to continue refining and finalizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> based on the previous comments. Additionally, I've included the SQL UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax supported by various vendors, including Dremio, Snowflake,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if a deeper discussion is needed. Let's keep moving forward, especially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sync, there was significant interest in storing UDFs in Iceberg and adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to better understand the new requirement of using UDFs for fine-grained
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> access control policies. This expands the use cases beyond just versioned
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and interoperable UDFs. Additionally, I learnt that many vendors are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interested in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations to be SQL. It merely does not specify (in this revision)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (adding a new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement of the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow only SQL representations, although it is certainly faviouring SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko Driesprong <fo...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concerns as this proposal focuses on SQL-based engines, while Python-based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> systems often work with data frames. Adding imperative languages like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Python would make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs are hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't fully understand yet how the proposed design addresses shareability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> making user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating this thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal and added comments. I think it would be helpful to also have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> design doc that covers the choices from the draft spec. For instance, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> choice to enumerate all possible function input struts rather than allowing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate function signatures is limiting. It would be nice to see a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion of the trade-offs and a rationale for the choice. I think it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would also be very helpful to have a few representative use cases for this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> included in the doc. That way the proposal can demonstrate that it solves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those use cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> where this is inconsistent with conventions in other specs. For example,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using string IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> model for spec versioning than the Iceberg view and table specs. It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requires readers to fail if there are any unknown fields, which prevents
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec from adding things that are fully backward-compatible. Other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg specs only require a version change to introduce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> forward-incompatible changes and I think that this should do the same to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow multiple function signatures per verison, but it is unclear how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encode them because a version is associated with a single function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax for creating functions across engines, so this doesn’t show that the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table-valued function shows a SELECT statement and it isn’t clear how this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If no more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interested, please have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is an interesting direction, but there might be some details that need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be fine tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> others who might be interested. Resharing since I do not think it was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> directly linked in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> since we didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review so far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attached in the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clarity to take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually meant scalar/aggregate/table user defined functions. Here are some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples of what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referred to is a variation of (1) where the API is data flow/data pipeline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API instead of SQL (e.g., Spark Scala). Yes, that is also possible in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> still explore some long term opportunities in this case. Consider you
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register a Spark temp view as some sort of data frame read, then it could
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> still be resolved to a Spark plan that is representable by an intermediate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representation. But I agree this gets very complicated very soon, and just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> having the case (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> note that a tabular SQL UDF can be used to build a parameterized view.  So,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there's definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> know more when you post the proposal, but I think this would be a very
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> difficult area to tackle across engines, languages, and memory models
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> without having a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initially supports SQL representations of UDFs (similar to views as shared
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the reference links above), the complexity involved will be similar to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Robert, and Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> publishing the draft spec (inspired by the view spec) this week to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> facilitate further discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different from the views? I feel we can say exactly the same thing for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg views, but yet we have Iceberg multi-dialect views implemented.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe it sounds like we are trying to draw a line between SQL vs other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> programming language as "code"? but I think SQL is just another type of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code, and we are already talking about compiling all these different code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dialects to an intermediate representation (using projects like Coral,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Substrait), which will be stored as another type of representation of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view. I think the same functionality can be used for UDFs if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding UDF support is a good idea, even just a multi-dialect one like view,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and that can allow engines to for example parse a view SQL, and when a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> define a common base upon which engines can build, so the argument that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs aren't practical, because engines are different, is probably only a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temporary concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> term, Iceberg should also try to tackle the idea to make views portable,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which is conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fan of adding a negative touch to the idea of having UDFs in Iceberg,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> especially not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about whether it's a good idea to add UDFs tracked by Iceberg catalogs. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think that Iceberg primarily deals with things that are centralized, like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables of data. While it would be great to have a common set of functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> across engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> know more when you post the proposal, but I think this would be a very
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> difficult area to tackle across engines, languages, and memory models
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> without having a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion to gauge the community interest in storing the Versioned SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> propose the spec addition for storing the versioned UDFs in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operate similarly to views in that they are associated with tables, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they can accept arguments and produce return values, or even function as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines like Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them in Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Interoperability between the engines. Potentially engines can understand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDFs written by other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> your thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi everyone,

The UDF spec has now been merged 🎉

Huge thanks to everyone involved for the thoughtful discussions, detailed
reviews, and steady progress over many iterations. This really wouldn’t
have been possible without the time, care, and collaboration from this
group.

Since the spec work is now complete, I will cancel the UDF spec sync
meeting moving forward. If new topics or follow up discussions come up, we
can always restart ad hoc meetings as needed.

Thanks again, and looking forward to the next steps (e.g., REST spec) and
reference implementations.

Yufei


On Tue, Jan 27, 2026 at 10:22 AM Yufei Gu <fl...@gmail.com> wrote:

> Thanks everyone for joining yesterday's UDF sync. We discussed a few
> things:
>
>    - Secure UDFs are a signal, not a guarantee. Security semantics are
>    engine specific, and administrators decide which engines are trusted. The
>    spec should avoid enforcing security behavior and clearly state it does not
>    provide security guarantees.
>    - Type definitions use Iceberg type JSON, are IDless, and allow nulls
>    everywhere. No required fields.
>    - Variant types are allowed and should be explicitly supported.
>    - SQL dialects stay coarse grained, for example spark or trino.
>    Version handling is an implementation detail.
>    - SQL UDFs first, Python UDFs later.
>    - The spec is close to final, with mostly wording cleanup remaining.
>
> The meeting recording is available here: https://youtu.be/HpFry5PlV1g
>
> Please take another look at PR
> https://github.com/apache/iceberg/pull/14117, and vote in the dev mailing
> thread.
>
> Yufei
>
>
> On Tue, Jan 13, 2026 at 12:14 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Thanks everyone for joining yesterday's UDF sync. We focused on next
>> steps for the global parameter name list and on how to approach default
>> values:
>> 1. We made good progress on parameter names by pushing down the parameter
>> name list to individual overloads and applying constraints on call
>> conventions.
>> 2. For default values, the current direction is to let engines decide how
>> they support them, rather than specifying default value semantics in the
>> UDF spec at this stage.
>>
>> The meeting recording is available here: https://youtu.be/2uKxYr1yX5w
>>
>> You can find detailed and corresponding changes in PR
>> https://github.com/apache/iceberg/pull/14117. The PR is ready for
>> another look. Feedback is very welcome.
>>
>> Yufei
>>
>>
>> On Mon, Jan 12, 2026 at 9:01 AM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi Folks,
>>>
>>> The UDF spec is overall a good sharp. Thanks for all the feedback. In
>>> today's sync, we could discuss the global parameter name list(drafted one
>>> google doc for its pros and cons,
>>> https://docs.google.com/document/d/1ipeEDTlgnBmoH_DJA3HWs6ZiLRaZGSDR-KwS__gc4-M/edit?tab=t.0)
>>> and whether we need to support the default value at this phase.
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Dec 15, 2025 at 2:18 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Thanks everyone for joining today’s UDF sync. We covered several topics:
>>>>
>>>> 1. We generally agreed to prefer a simpler form for complex function
>>>> type signatures, instead of the Iceberg type JSON format (
>>>> https://iceberg.apache.org/spec/#appendix-c-json-serialization).
>>>> 2. We discussed the motivations and concerns around having a global
>>>> parameter name list. Given the trade-offs, we decided to take a deeper look
>>>> offline.
>>>> 3. We talked about default values in terms of overload disambiguation
>>>> and cross-engine support, and agreed that this needs further investigation.
>>>>
>>>> Going forward, we’ll use the following Google Doc to record key design
>>>> decisions and rationale, so we have a shared reference as discussions
>>>> evolve:
>>>>
>>>> https://docs.google.com/document/d/1GKtqFt9gnlJxwo5kfExupDf1jASlgcnDeZv8vEaJQUs/edit?usp=sharing
>>>>
>>>> The meeting recording is available here: https://youtu.be/tF7Ur88snR0
>>>>
>>>> Thanks Kevin for the proposal. Given the holiday season, we'd like to
>>>> propose cancelling the next Iceberg UDF Sync scheduled for December 29,
>>>> 2025. Please let me know if there are any objections.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Fri, Dec 5, 2025 at 8:51 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Thanks for joining Monday’s UDF sync!
>>>>> We covered two updates:
>>>>>
>>>>>    - Parameter names moved to the top level, this keeps
>>>>>    named-argument invocation consistent across overloads.
>>>>>    - Default value support added, callers can now omit parameters
>>>>>    that have defaults.
>>>>>
>>>>> The recording is available here for more details:
>>>>> https://youtu.be/6CBMwwbISOE
>>>>>
>>>>> I’ve also added more examples demonstrating default values. Please
>>>>> take a look at the PR: https://github.com/apache/iceberg/pull/14117
>>>>>
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Wed, Nov 26, 2025 at 5:57 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> Made a few changes to the spec. The PR(
>>>>>> https://github.com/apache/iceberg/pull/14117) is ready for another
>>>>>> look.
>>>>>> 1. Moved the parameter name list to the top level, so that
>>>>>> named-argument invocation is consistent across overloads.
>>>>>> 2. Added support for default value, which allows caller to ignore
>>>>>> parameters with default values.
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 17, 2025 at 6:42 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap
>>>>>>> of what we discussed and agreed on:
>>>>>>>
>>>>>>>    1. Huaxin walked us through the Spark POC(
>>>>>>>    https://github.com/apache/iceberg/pull/14505). Thanks Huaxin!
>>>>>>>    2. When default values are enabled, ambiguous overloads must not
>>>>>>>    be allowed. We agreed that definition-id alone is insufficient for conflict
>>>>>>>    detection; additional signature-level check across overloads is needed.
>>>>>>>    3. We will explore the idea that parameter names are globally
>>>>>>>    consistent across multiple overloads. For example, if the overload foo(int
>>>>>>>    x, int y) exists, foo(int a) is not allowed, but foo(int x) is OK. We could
>>>>>>>    put a list of names and their docs at the top level. This benefits UX and
>>>>>>>    may potentially benefit the strategy to pick up the overload to use given
>>>>>>>    the inputs. We may bring back the parameter names at the representation
>>>>>>>    level, so that it's more flexible.
>>>>>>>    4. Should an engine ignore the default values or fail fast if it
>>>>>>>    doesn’t support it? We concluded that this is up to the engines, and
>>>>>>>    engines could enforce all parameters to be provided when default values
>>>>>>>    aren't supported.
>>>>>>>    5. How do we give the default value for complex types(e.g.
>>>>>>>    struct)? We should follow the way table spec handles the default value(
>>>>>>>    https://iceberg.apache.org/spec/#edge-interpolation-algorithm).
>>>>>>>
>>>>>>> You can watch the recording here for more details:
>>>>>>> https://youtu.be/Wxz1nSP48k0
>>>>>>>
>>>>>>> Please take another look at PR
>>>>>>> https://github.com/apache/iceberg/pull/14117 when you get a chance.
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Nov 14, 2025 at 4:29 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> The PR (https://github.com/apache/iceberg/pull/14117) has been
>>>>>>>> updated, please take another look.
>>>>>>>> For next Monday’s UDF sync, Huaxin will demo a Spark POC showing
>>>>>>>> how the current (WIP) UDF spec works. We can also walk through how default
>>>>>>>> values should be introduced into the spec.
>>>>>>>>
>>>>>>>> *TL;DR:* I haven’t added default-value support in the PR yet. It’s
>>>>>>>> turning out to be trickier than expected because default values make
>>>>>>>> parameters optional. That introduces ambiguity for overloads like:
>>>>>>>>
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    foo(int)
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    foo(int, int default 0)
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    foo(int, float default 1.0)
>>>>>>>>
>>>>>>>> All of these could be invoked as foo(int). The first question is:
>>>>>>>> should we allow such overloads to coexist? Probably not, both Postgres and
>>>>>>>> Snowflake support default values, but neither allows this form of ambiguous
>>>>>>>> overloading.
>>>>>>>> Assuming we disallow it, the current UDF definition-id (type tuple)
>>>>>>>> isn’t sufficient to detect conflicts. We also need to check the signatures
>>>>>>>> of existing overloads to determine whether a new one is valid. For example,
>>>>>>>> if (int, int) exists, then neither (int) nor (int, float) should be
>>>>>>>> allowed.
>>>>>>>>
>>>>>>>> Reference:
>>>>>>>> Postgres behavior on default parameters:
>>>>>>>> https://www.postgresql.org/docs/current/static/typeconv-func.html
>>>>>>>> Functions with default arguments match calls that omit zero or more
>>>>>>>> defaultable parameters. If multiple functions match, the earliest in the
>>>>>>>> search path is chosen; if they are equally specific, Postgres throws an
>>>>>>>> “ambiguous function call” error.
>>>>>>>> Thanks,
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> Thanks for joining today’s community UDF sync! Here’s a quick
>>>>>>>>> recap of what we discussed and agreed on:
>>>>>>>>> 1. We’ll keep parameters only at the definition layer to support
>>>>>>>>> named-argument invocation without extra work from the engine side.
>>>>>>>>> 2. We agreed to add support for default parameter values.
>>>>>>>>> 3. Moving forward with a flag for nullable vs. non-nullable return
>>>>>>>>> values, treating it as a hint rather than a strict rule.
>>>>>>>>> 4. The udf metadata will preserve the original case of parameter
>>>>>>>>> names, and engines can decide whether to normalize or convert them.
>>>>>>>>>
>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>> https://www.youtube.com/watch?v=0i6nKfDNdGw
>>>>>>>>> Sorry that today's sync wasn't in the Iceberg Dev Event
>>>>>>>>> calendar(It was removed somehow). I have added it back to Iceberg Dev Event
>>>>>>>>> as a biweekly sync. Let me know if you have any questions.
>>>>>>>>>
>>>>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>>> for item 1 and 2 soon. The overall content will remain largely the same.
>>>>>>>>> Please take a look when you get a chance!
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Ryan and me are in this meeting, please join
>>>>>>>>>>
>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>> Monday, November 3 · 9:00 – 10:00am
>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>> Google Meet joining info
>>>>>>>>>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi folks,
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot for the review! Made a few changes to the spec:
>>>>>>>>>>> 1. Renamed "overload" to "definition"
>>>>>>>>>>> 2. Changed definition id from long to string with the format of
>>>>>>>>>>> parameter type tuple.
>>>>>>>>>>> 3. Explicitly claimed that named argument invocation is supported
>>>>>>>>>>> 4. Added a nullable flag for return values
>>>>>>>>>>> 5. Added a few rules for inputs and returns
>>>>>>>>>>>
>>>>>>>>>>> Please find more details in the PR(
>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117). Here is a doc
>>>>>>>>>>> comparing engine behaviors in terms of named argument invocation, nullable
>>>>>>>>>>> return value, default values for inputs, null handling:
>>>>>>>>>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>>>>>>>>>> This can help for decision making.
>>>>>>>>>>>
>>>>>>>>>>> We could discuss the following items in Monday's meeting:
>>>>>>>>>>> 1. Whether we need to push down parameters to representation
>>>>>>>>>>> given we want to support named argument invocation, it requires extra
>>>>>>>>>>> mapping on the engine side.
>>>>>>>>>>> 2. Whether we support default values.  Spark plans to support
>>>>>>>>>>> it, Snowflake supports it already.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for joining today’s community UDF sync. Here’s a summary
>>>>>>>>>>>> of what we discussed and agreed upon:
>>>>>>>>>>>>
>>>>>>>>>>>>    1. We agreed to use Iceberg types for each field in the UDF
>>>>>>>>>>>>    spec. This approach ensures consistency with the table spec and allows for
>>>>>>>>>>>>    straightforward conversion to JSON types.
>>>>>>>>>>>>    2. Using numeric overload IDs can have side effects, for
>>>>>>>>>>>>    instance, deleting an overload and later creating a new one with the same
>>>>>>>>>>>>    signature will use a different numeric id, which points to the same
>>>>>>>>>>>>    overload. The consensus is to represent the overload ID as a string derived
>>>>>>>>>>>>    from parameter types and their order.
>>>>>>>>>>>>    3. Parameter names must remain consistent across all
>>>>>>>>>>>>    overload versions. We agreed to add explicit rules:
>>>>>>>>>>>>       1. Parameter names must match across all overload
>>>>>>>>>>>>       versions.
>>>>>>>>>>>>       2. Renaming a parameter requires removing all previous
>>>>>>>>>>>>       versions that use the old name.
>>>>>>>>>>>>    4. We discussed whether parameters or return types can be
>>>>>>>>>>>>    nullable and how engines should handle null values. We’ll research SQL
>>>>>>>>>>>>    standards and common engine behaviors before finalizing. The current
>>>>>>>>>>>>    leaning is:
>>>>>>>>>>>>       1. Allow all parameters to be nullable.
>>>>>>>>>>>>       2. Allow the return type to include an explicit
>>>>>>>>>>>>       nullability annotation.
>>>>>>>>>>>>
>>>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>>>> https://youtu.be/IUd-SN0CQRs
>>>>>>>>>>>> Will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>>>>>> soon.
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks a lot for joining Monday's UDF sync. Here is the
>>>>>>>>>>>>> summary:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1. Replace “definitions version” with “definitions logs”,
>>>>>>>>>>>>>    which record overload changes and timestamps for informational purposes
>>>>>>>>>>>>>    only.
>>>>>>>>>>>>>    2. Use a globally increasing numeric ID for overloads
>>>>>>>>>>>>>    instead of UUIDs.
>>>>>>>>>>>>>    3. Add explicit types to all fields in the spec to clarify
>>>>>>>>>>>>>    the metadata structure.
>>>>>>>>>>>>>    4. Add null-handling optimization hints to help engines
>>>>>>>>>>>>>    skip evaluation when inputs are null. (Thanks, Talat!)
>>>>>>>>>>>>>    5. Function parameters remain ordered, but we don’t need
>>>>>>>>>>>>>    to introduce IDs for each parameter. We’ll double-check the Java
>>>>>>>>>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>>>>>>>>>    nested structs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>>>>> https://youtu.be/gJnueQODWvs
>>>>>>>>>>>>> I have updated the PR (
>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) for the above
>>>>>>>>>>>>> feedback. Please take a look and let me know if you have any questions.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks a lot for joining today's UDF sync. Here is the
>>>>>>>>>>>>>> summary:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    1. Instead of relying on dynamic inference, the return
>>>>>>>>>>>>>>    table’s schema for user-defined table functions (UDTFs) should be
>>>>>>>>>>>>>>    explicitly defined.
>>>>>>>>>>>>>>    2. Whether a function is a UDTF should be captured as a
>>>>>>>>>>>>>>    dedicated attribute, rather than being inferred indirectly from the return
>>>>>>>>>>>>>>    type.
>>>>>>>>>>>>>>    3. The interpretation of a UDF body (whether it is
>>>>>>>>>>>>>>    treated as a partial SQL expression or as a full SELECT statement) should
>>>>>>>>>>>>>>    be determined by engines. Example: `SELECT x +1` vs. `x + 1`. Different
>>>>>>>>>>>>>>    engines have different takes.
>>>>>>>>>>>>>>    4. User-defined aggregation functions (UDAFs) are out of
>>>>>>>>>>>>>>    scope for now.
>>>>>>>>>>>>>>    5. Each overload should include its own current-version
>>>>>>>>>>>>>>    field. This avoids relying solely on the global `definition-versions` when
>>>>>>>>>>>>>>    querying the current version of one overload.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>>>>>>>>>> I will update the PR(
>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) shortly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <
>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Really appreciated feedback from you all over the past few
>>>>>>>>>>>>>>> months. I've filed the initial PR for the UDF spec:
>>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures
>>>>>>>>>>>>>>> the consensus we've built and addresses the write amplification concern
>>>>>>>>>>>>>>> raised in our last discussion.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please take a look and share your thoughts. Happy to discuss
>>>>>>>>>>>>>>> it further during Monday's meeting as well.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We covered the UDF metadata structure, captured in this
>>>>>>>>>>>>>>>> doc:
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We also discussed a way to avoid copying every overload
>>>>>>>>>>>>>>>> into the new metadata JSON when creating a new version. One of ideas is to
>>>>>>>>>>>>>>>> introduce a global version array, this is not yet reflected in the doc, but
>>>>>>>>>>>>>>>> I’ll update it shortly. Other key points:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>>>>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>>>>>>>>>>    execute.
>>>>>>>>>>>>>>>>    - Keeping the version while referring to an UDF
>>>>>>>>>>>>>>>>    probably isn't a good idea. Users are responsible for updating downstream
>>>>>>>>>>>>>>>>    views if they reference older UDF versions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In
>>>>>>>>>>>>>>>>> general, we discussed the UDF metadata structure, captured at this doc(
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    1. Each UDF overload has its own return type. e.g.,
>>>>>>>>>>>>>>>>>    `add(int, int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>>>>>>>>>    2. Return type should be explicitly specified, no
>>>>>>>>>>>>>>>>>    implicit or statement-based return type inference should be allowed.
>>>>>>>>>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>>>>>>>>>    properties at the overload level.
>>>>>>>>>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>>>>>>>>>    5. Introducing a dedicated signature definitions
>>>>>>>>>>>>>>>>>    section to centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>>>>>>>>>    6. Whether to have versioned open properties or not.
>>>>>>>>>>>>>>>>>    Versioned properties can lead to unnecessary copying of a bag of properties
>>>>>>>>>>>>>>>>>    into each version, while it provides a clear history of properties for any
>>>>>>>>>>>>>>>>>    future debugging and understanding of the UDF behavior at a
>>>>>>>>>>>>>>>>>    specific point in time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <
>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi everyone, here’s the summary from our last sync on
>>>>>>>>>>>>>>>>>> 8/11. Apologies for the delay!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>>>>>>>>>       - We agreed to combine overloads with the same
>>>>>>>>>>>>>>>>>>       name into a single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>>>>>>>>>       - Listing UDFs will return a list of UDF names,
>>>>>>>>>>>>>>>>>>       not a list of individual signatures.
>>>>>>>>>>>>>>>>>>       - Loading a UDF by name will return all of its
>>>>>>>>>>>>>>>>>>       overloads.
>>>>>>>>>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>>>>>>>>>       - A global version number will track changes
>>>>>>>>>>>>>>>>>>       across the entire UDF entity, it increments monolithically.
>>>>>>>>>>>>>>>>>>       - Each overload will also maintain its own version
>>>>>>>>>>>>>>>>>>       (e.g., updated_at_version) to trace changes specific to that overload.
>>>>>>>>>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>>>>>>>>>    this stage.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <
>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs
>>>>>>>>>>>>>>>>>>> with multiple signatures under the same name, which can serve both
>>>>>>>>>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Per my investigation[1], most engines support
>>>>>>>>>>>>>>>>>>> overloading by arguments and allow implicit conversions like numeric
>>>>>>>>>>>>>>>>>>> widening (e.g., INT → BIGINT/FLOAT). The resolution approach causes issues
>>>>>>>>>>>>>>>>>>> like silent behavior change. Here is an example:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected
>>>>>>>>>>>>>>>>>>>    code.
>>>>>>>>>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>>>>>>>>>    - Engine’s best-match resolution now binds the same
>>>>>>>>>>>>>>>>>>>    call to the new overload, changing behavior without modifying the query.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To mitigate this issue, we have to choose between these
>>>>>>>>>>>>>>>>>>> two access control models:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all
>>>>>>>>>>>>>>>>>>>    overloads of a function name.
>>>>>>>>>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to
>>>>>>>>>>>>>>>>>>>    specific signatures.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The general recommendation is to adopt *Model A.* It
>>>>>>>>>>>>>>>>>>> trades some precision for safety and simplicity, while eliminating the
>>>>>>>>>>>>>>>>>>> silent behavior change problem. More details are in this doc[1].
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>> We have discussed how to define function identifiers
>>>>>>>>>>>>>>>>>>>> (should also handle function overloading). Ryan suggested that we should
>>>>>>>>>>>>>>>>>>>> check how Spark does it. We can refer to functions using an identifier and
>>>>>>>>>>>>>>>>>>>> then bind the different signatures to it. So that access policies can be
>>>>>>>>>>>>>>>>>>>> applied per identifier. This is also linked to how we want to version the
>>>>>>>>>>>>>>>>>>>> functions when overloading is supported.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I will check more about this and update the proposal
>>>>>>>>>>>>>>>>>>>> doc.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please check/subscribe to the dev events calendar for
>>>>>>>>>>>>>>>>>>>> the next meeting link (Aug 11).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg
>>>>>>>>>>>>>>>>>>>>> Dev Events" calendar for tomorrow 7/28 at 9AM PT. I
>>>>>>>>>>>>>>>>>>>>> missed the last one, but i'll be at this one.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> No one joined the sync today. I came to know that
>>>>>>>>>>>>>>>>>>>>>> Yufei is on holiday, and Ryan and others couldn't make it, similar to the
>>>>>>>>>>>>>>>>>>>>>> last sync. It seems Yufei might have forgotten to transfer meeting
>>>>>>>>>>>>>>>>>>>>>> ownership as well, as new members needed admin approval and couldn't join
>>>>>>>>>>>>>>>>>>>>>> automatically this week. Also, I can understand it is summer holiday season
>>>>>>>>>>>>>>>>>>>>>> for many.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I've updated the function signature schema and other
>>>>>>>>>>>>>>>>>>>>>> open points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a
>>>>>>>>>>>>>>>>>>>>>>> top level field to a field inside “version” along with a representation, so
>>>>>>>>>>>>>>>>>>>>>>> that properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> That said, it's still an open question whether we
>>>>>>>>>>>>>>>>>>>>>>> need any non-versioned properties. We can introduce them later if a use
>>>>>>>>>>>>>>>>>>>>>>> case arises.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from
>>>>>>>>>>>>>>>>>>>>>>>> the representation history. Here are reasons:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    1. Each version still enforces a single
>>>>>>>>>>>>>>>>>>>>>>>>    signature. Although the signatures array is global to the UDF, each version
>>>>>>>>>>>>>>>>>>>>>>>>    references just one signature ID. Rollbacks to historical versions remain
>>>>>>>>>>>>>>>>>>>>>>>>    safe.
>>>>>>>>>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types,
>>>>>>>>>>>>>>>>>>>>>>>>    they should remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>> We have discussed the action items from the last
>>>>>>>>>>>>>>>>>>>>>>>>> sync (*see Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of
>>>>>>>>>>>>>>>>>>>>>>>>>    the engines and in the roadmaps of many engines. Iceberg will support it.
>>>>>>>>>>>>>>>>>>>>>>>>>    We will maintain the `FunctionIdentifier` (extends `TableIdentifer` but
>>>>>>>>>>>>>>>>>>>>>>>>>    also have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in
>>>>>>>>>>>>>>>>>>>>>>>>>    a bag, we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural
>>>>>>>>>>>>>>>>>>>>>>>>>    bodies are supported by some engines. Iceberg will support it. Store the
>>>>>>>>>>>>>>>>>>>>>>>>>    body as it is while creating function by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    - Standardizing the property names
>>>>>>>>>>>>>>>>>>>>>>>>>    (deterministic, secure).
>>>>>>>>>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level
>>>>>>>>>>>>>>>>>>>>>>>>>    replace is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>>>>>>>>>    - Signature should be associated with
>>>>>>>>>>>>>>>>>>>>>>>>>    representation?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please
>>>>>>>>>>>>>>>>>>>>>>>>>    review the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the
>>>>>>>>>>>>>>>>>>>>>>>>>> whole metadata is encrypted, we don't have to worry about just hiding the
>>>>>>>>>>>>>>>>>>>>>>>>>> UDF body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling
>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown are required.We will need a named key(e.g., secure) somewhere, no
>>>>>>>>>>>>>>>>>>>>>>>>>>> matter if it is a top level property or a key as a part of the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>> properties. So that both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec would require? Would it require hiding the UDF definition from
>>>>>>>>>>>>>>>>>>>>>>>>>>>> users and require specific pushdown cases be disabled? The use cases seem
>>>>>>>>>>>>>>>>>>>>>>>>>>>> valid, but I'm trying to understand the requirements this places on engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and why it needs to be part of the spec, rather than part of the properties
>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the UDF.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    concealing the UDF body and details like the list of imports, some of them
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    aren’t applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    in an isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    For example, secure UDFs may disable certain optimizations—such as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    predicate pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that the secure flag is primarily an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instruction to the engine to behave differently. While it's largely an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine-side behavior, we still need to include this flag in the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition to indicate whether a UDF is secure, especially considering the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> perf penalty introduced by scenario #3. We should clearly recommend that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users avoid marking UDFs as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supporting a "secure" UDF? What use case are you addressing and what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specifically changes about how the UDF is handled? If the idea is to hide
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDF definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine". When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but whether those statements run within a single transaction should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> treated as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “secure UDF” option supported by some engines[1], so the body and any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sensitive details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    syntax supported by different engines (Snowflake, databricks, Dremio,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Trino, OSS spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    separator, like $$ or '' or none. Action item was to check whether engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    support multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Need to check whether these engines support function overloading for SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDFs. Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    discussed the deterministic flag and concluded that we keep the independent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    fields (like deterministic) in spec only if the majority of engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    supports it. Else it will be passed in a property bag (engine specific).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    And it is the engine's responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> document here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    the majority agreed *not to* (see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    recording for details).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning —
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    it will be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    each catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    existing table/view spec — e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    location will remain *required* for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    document
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    vote once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> community sync for the UDF project. Everyone’s welcome to drop in and share
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ideas! Here is the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun.zou to discuss the UDF proposal. We covered several key points, though
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some are still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines may not support vararg syntax in CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION, Iceberg UDFs could represent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently doesn’t support generic types (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> object), we can only map
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Python as a language for SQL UDFs seems promising, especially given its
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> potential to resolve interoperability challenges. Some engines, however,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> require platform version and package dependency details to execute Python
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code—this should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> two primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (open to more interested participants) to continue refining and finalizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> based on the previous comments. Additionally, I've included the SQL UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax supported by various vendors, including Dremio, Snowflake,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if a deeper discussion is needed. Let's keep moving forward, especially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sync, there was significant interest in storing UDFs in Iceberg and adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to better understand the new requirement of using UDFs for fine-grained
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> access control policies. This expands the use cases beyond just versioned
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and interoperable UDFs. Additionally, I learnt that many vendors are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interested in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations to be SQL. It merely does not specify (in this revision)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (adding a new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement of the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko Driesprong <fo...@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concerns as this proposal focuses on SQL-based engines, while Python-based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> systems often work with data frames. Adding imperative languages like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Python would make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs are hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't fully understand yet how the proposed design addresses shareability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> making user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal and added comments. I think it would be helpful to also have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> design doc that covers the choices from the draft spec. For instance, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> choice to enumerate all possible function input struts rather than allowing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate function signatures is limiting. It would be nice to see a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion of the trade-offs and a rationale for the choice. I think it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would also be very helpful to have a few representative use cases for this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> included in the doc. That way the proposal can demonstrate that it solves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those use cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> where this is inconsistent with conventions in other specs. For example,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using string IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> model for spec versioning than the Iceberg view and table specs. It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requires readers to fail if there are any unknown fields, which prevents
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec from adding things that are fully backward-compatible. Other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg specs only require a version change to introduce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> forward-incompatible changes and I think that this should do the same to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow multiple function signatures per verison, but it is unclear how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encode them because a version is associated with a single function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax for creating functions across engines, so this doesn’t show that the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table-valued function shows a SELECT statement and it isn’t clear how this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If no more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interested, please have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is an interesting direction, but there might be some details that need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be fine tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> others who might be interested. Resharing since I do not think it was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> directly linked in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> since we didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review so far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attached in the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clarity to take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> meant scalar/aggregate/table user defined functions. Here are some examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referred to is a variation of (1) where the API is data flow/data pipeline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API instead of SQL (e.g., Spark Scala). Yes, that is also possible in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explore some long term opportunities in this case. Consider you register a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark temp view as some sort of data frame read, then it could still be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolved to a Spark plan that is representable by an intermediate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representation. But I agree this gets very complicated very soon, and just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> having the case (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> note that a tabular SQL UDF can be used to build a parameterized view.  So,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there's definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initially supports SQL representations of UDFs (similar to views as shared
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the reference links above), the complexity involved will be similar to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Robert, and Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> publishing the draft spec (inspired by the view spec) this week to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> facilitate further discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different from the views? I feel we can say exactly the same thing for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg views, but yet we have Iceberg multi-dialect views implemented.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe it sounds like we are trying to draw a line between SQL vs other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> programming language as "code"? but I think SQL is just another type of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code, and we are already talking about compiling all these different code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dialects to an intermediate representation (using projects like Coral,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Substrait), which will be stored as another type of representation of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view. I think the same functionality can be used for UDFs if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding UDF support is a good idea, even just a multi-dialect one like view,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and that can allow engines to for example parse a view SQL, and when a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> define a common base upon which engines can build, so the argument that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs aren't practical, because engines are different, is probably only a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temporary concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of adding a negative touch to the idea of having UDFs in Iceberg,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> especially not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about whether it's a good idea to add UDFs tracked by Iceberg catalogs. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think that Iceberg primarily deals with things that are centralized, like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables of data. While it would be great to have a common set of functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> across engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> know more when you post the proposal, but I think this would be a very
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> difficult area to tackle across engines, languages, and memory models
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> without having a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion to gauge the community interest in storing the Versioned SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> propose the spec addition for storing the versioned UDFs in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operate similarly to views in that they are associated with tables, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they can accept arguments and produce return values, or even function as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines like Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Interoperability between the engines. Potentially engines can understand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDFs written by other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Thanks everyone for joining yesterday's UDF sync. We discussed a few things:

   - Secure UDFs are a signal, not a guarantee. Security semantics are
   engine specific, and administrators decide which engines are trusted. The
   spec should avoid enforcing security behavior and clearly state it does not
   provide security guarantees.
   - Type definitions use Iceberg type JSON, are IDless, and allow nulls
   everywhere. No required fields.
   - Variant types are allowed and should be explicitly supported.
   - SQL dialects stay coarse grained, for example spark or trino. Version
   handling is an implementation detail.
   - SQL UDFs first, Python UDFs later.
   - The spec is close to final, with mostly wording cleanup remaining.

The meeting recording is available here: https://youtu.be/HpFry5PlV1g

Please take another look at PR https://github.com/apache/iceberg/pull/14117,
and vote in the dev mailing thread.

Yufei


On Tue, Jan 13, 2026 at 12:14 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks everyone for joining yesterday's UDF sync. We focused on next steps
> for the global parameter name list and on how to approach default values:
> 1. We made good progress on parameter names by pushing down the parameter
> name list to individual overloads and applying constraints on call
> conventions.
> 2. For default values, the current direction is to let engines decide how
> they support them, rather than specifying default value semantics in the
> UDF spec at this stage.
>
> The meeting recording is available here: https://youtu.be/2uKxYr1yX5w
>
> You can find detailed and corresponding changes in PR
> https://github.com/apache/iceberg/pull/14117. The PR is ready for another
> look. Feedback is very welcome.
>
> Yufei
>
>
> On Mon, Jan 12, 2026 at 9:01 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi Folks,
>>
>> The UDF spec is overall a good sharp. Thanks for all the feedback. In
>> today's sync, we could discuss the global parameter name list(drafted one
>> google doc for its pros and cons,
>> https://docs.google.com/document/d/1ipeEDTlgnBmoH_DJA3HWs6ZiLRaZGSDR-KwS__gc4-M/edit?tab=t.0)
>> and whether we need to support the default value at this phase.
>>
>> Yufei
>>
>>
>> On Mon, Dec 15, 2025 at 2:18 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Thanks everyone for joining today’s UDF sync. We covered several topics:
>>>
>>> 1. We generally agreed to prefer a simpler form for complex function
>>> type signatures, instead of the Iceberg type JSON format (
>>> https://iceberg.apache.org/spec/#appendix-c-json-serialization).
>>> 2. We discussed the motivations and concerns around having a global
>>> parameter name list. Given the trade-offs, we decided to take a deeper look
>>> offline.
>>> 3. We talked about default values in terms of overload disambiguation
>>> and cross-engine support, and agreed that this needs further investigation.
>>>
>>> Going forward, we’ll use the following Google Doc to record key design
>>> decisions and rationale, so we have a shared reference as discussions
>>> evolve:
>>>
>>> https://docs.google.com/document/d/1GKtqFt9gnlJxwo5kfExupDf1jASlgcnDeZv8vEaJQUs/edit?usp=sharing
>>>
>>> The meeting recording is available here: https://youtu.be/tF7Ur88snR0
>>>
>>> Thanks Kevin for the proposal. Given the holiday season, we'd like to
>>> propose cancelling the next Iceberg UDF Sync scheduled for December 29,
>>> 2025. Please let me know if there are any objections.
>>>
>>> Yufei
>>>
>>>
>>> On Fri, Dec 5, 2025 at 8:51 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Thanks for joining Monday’s UDF sync!
>>>> We covered two updates:
>>>>
>>>>    - Parameter names moved to the top level, this keeps named-argument
>>>>    invocation consistent across overloads.
>>>>    - Default value support added, callers can now omit parameters that
>>>>    have defaults.
>>>>
>>>> The recording is available here for more details:
>>>> https://youtu.be/6CBMwwbISOE
>>>>
>>>> I’ve also added more examples demonstrating default values. Please take
>>>> a look at the PR: https://github.com/apache/iceberg/pull/14117
>>>>
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Wed, Nov 26, 2025 at 5:57 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Made a few changes to the spec. The PR(
>>>>> https://github.com/apache/iceberg/pull/14117) is ready for another
>>>>> look.
>>>>> 1. Moved the parameter name list to the top level, so that
>>>>> named-argument invocation is consistent across overloads.
>>>>> 2. Added support for default value, which allows caller to ignore
>>>>> parameters with default values.
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Mon, Nov 17, 2025 at 6:42 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap
>>>>>> of what we discussed and agreed on:
>>>>>>
>>>>>>    1. Huaxin walked us through the Spark POC(
>>>>>>    https://github.com/apache/iceberg/pull/14505). Thanks Huaxin!
>>>>>>    2. When default values are enabled, ambiguous overloads must not
>>>>>>    be allowed. We agreed that definition-id alone is insufficient for conflict
>>>>>>    detection; additional signature-level check across overloads is needed.
>>>>>>    3. We will explore the idea that parameter names are globally
>>>>>>    consistent across multiple overloads. For example, if the overload foo(int
>>>>>>    x, int y) exists, foo(int a) is not allowed, but foo(int x) is OK. We could
>>>>>>    put a list of names and their docs at the top level. This benefits UX and
>>>>>>    may potentially benefit the strategy to pick up the overload to use given
>>>>>>    the inputs. We may bring back the parameter names at the representation
>>>>>>    level, so that it's more flexible.
>>>>>>    4. Should an engine ignore the default values or fail fast if it
>>>>>>    doesn’t support it? We concluded that this is up to the engines, and
>>>>>>    engines could enforce all parameters to be provided when default values
>>>>>>    aren't supported.
>>>>>>    5. How do we give the default value for complex types(e.g.
>>>>>>    struct)? We should follow the way table spec handles the default value(
>>>>>>    https://iceberg.apache.org/spec/#edge-interpolation-algorithm).
>>>>>>
>>>>>> You can watch the recording here for more details:
>>>>>> https://youtu.be/Wxz1nSP48k0
>>>>>>
>>>>>> Please take another look at PR
>>>>>> https://github.com/apache/iceberg/pull/14117 when you get a chance.
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Fri, Nov 14, 2025 at 4:29 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> The PR (https://github.com/apache/iceberg/pull/14117) has been
>>>>>>> updated, please take another look.
>>>>>>> For next Monday’s UDF sync, Huaxin will demo a Spark POC showing how
>>>>>>> the current (WIP) UDF spec works. We can also walk through how default
>>>>>>> values should be introduced into the spec.
>>>>>>>
>>>>>>> *TL;DR:* I haven’t added default-value support in the PR yet. It’s
>>>>>>> turning out to be trickier than expected because default values make
>>>>>>> parameters optional. That introduces ambiguity for overloads like:
>>>>>>>
>>>>>>>    -
>>>>>>>
>>>>>>>    foo(int)
>>>>>>>    -
>>>>>>>
>>>>>>>    foo(int, int default 0)
>>>>>>>    -
>>>>>>>
>>>>>>>    foo(int, float default 1.0)
>>>>>>>
>>>>>>> All of these could be invoked as foo(int). The first question is:
>>>>>>> should we allow such overloads to coexist? Probably not, both Postgres and
>>>>>>> Snowflake support default values, but neither allows this form of ambiguous
>>>>>>> overloading.
>>>>>>> Assuming we disallow it, the current UDF definition-id (type tuple)
>>>>>>> isn’t sufficient to detect conflicts. We also need to check the signatures
>>>>>>> of existing overloads to determine whether a new one is valid. For example,
>>>>>>> if (int, int) exists, then neither (int) nor (int, float) should be
>>>>>>> allowed.
>>>>>>>
>>>>>>> Reference:
>>>>>>> Postgres behavior on default parameters:
>>>>>>> https://www.postgresql.org/docs/current/static/typeconv-func.html
>>>>>>> Functions with default arguments match calls that omit zero or more
>>>>>>> defaultable parameters. If multiple functions match, the earliest in the
>>>>>>> search path is chosen; if they are equally specific, Postgres throws an
>>>>>>> “ambiguous function call” error.
>>>>>>> Thanks,
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap
>>>>>>>> of what we discussed and agreed on:
>>>>>>>> 1. We’ll keep parameters only at the definition layer to support
>>>>>>>> named-argument invocation without extra work from the engine side.
>>>>>>>> 2. We agreed to add support for default parameter values.
>>>>>>>> 3. Moving forward with a flag for nullable vs. non-nullable return
>>>>>>>> values, treating it as a hint rather than a strict rule.
>>>>>>>> 4. The udf metadata will preserve the original case of parameter
>>>>>>>> names, and engines can decide whether to normalize or convert them.
>>>>>>>>
>>>>>>>> You can watch the recording here for more details:
>>>>>>>> https://www.youtube.com/watch?v=0i6nKfDNdGw
>>>>>>>> Sorry that today's sync wasn't in the Iceberg Dev Event calendar(It
>>>>>>>> was removed somehow). I have added it back to Iceberg Dev Event as a
>>>>>>>> biweekly sync. Let me know if you have any questions.
>>>>>>>>
>>>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>> for item 1 and 2 soon. The overall content will remain largely the same.
>>>>>>>> Please take a look when you get a chance!
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ryan and me are in this meeting, please join
>>>>>>>>>
>>>>>>>>> Iceberg UDF sync
>>>>>>>>> Monday, November 3 · 9:00 – 10:00am
>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>> Google Meet joining info
>>>>>>>>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> Thanks a lot for the review! Made a few changes to the spec:
>>>>>>>>>> 1. Renamed "overload" to "definition"
>>>>>>>>>> 2. Changed definition id from long to string with the format of
>>>>>>>>>> parameter type tuple.
>>>>>>>>>> 3. Explicitly claimed that named argument invocation is supported
>>>>>>>>>> 4. Added a nullable flag for return values
>>>>>>>>>> 5. Added a few rules for inputs and returns
>>>>>>>>>>
>>>>>>>>>> Please find more details in the PR(
>>>>>>>>>> https://github.com/apache/iceberg/pull/14117). Here is a doc
>>>>>>>>>> comparing engine behaviors in terms of named argument invocation, nullable
>>>>>>>>>> return value, default values for inputs, null handling:
>>>>>>>>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>>>>>>>>> This can help for decision making.
>>>>>>>>>>
>>>>>>>>>> We could discuss the following items in Monday's meeting:
>>>>>>>>>> 1. Whether we need to push down parameters to representation
>>>>>>>>>> given we want to support named argument invocation, it requires extra
>>>>>>>>>> mapping on the engine side.
>>>>>>>>>> 2. Whether we support default values.  Spark plans to support it,
>>>>>>>>>> Snowflake supports it already.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi folks,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for joining today’s community UDF sync. Here’s a summary
>>>>>>>>>>> of what we discussed and agreed upon:
>>>>>>>>>>>
>>>>>>>>>>>    1. We agreed to use Iceberg types for each field in the UDF
>>>>>>>>>>>    spec. This approach ensures consistency with the table spec and allows for
>>>>>>>>>>>    straightforward conversion to JSON types.
>>>>>>>>>>>    2. Using numeric overload IDs can have side effects, for
>>>>>>>>>>>    instance, deleting an overload and later creating a new one with the same
>>>>>>>>>>>    signature will use a different numeric id, which points to the same
>>>>>>>>>>>    overload. The consensus is to represent the overload ID as a string derived
>>>>>>>>>>>    from parameter types and their order.
>>>>>>>>>>>    3. Parameter names must remain consistent across all
>>>>>>>>>>>    overload versions. We agreed to add explicit rules:
>>>>>>>>>>>       1. Parameter names must match across all overload
>>>>>>>>>>>       versions.
>>>>>>>>>>>       2. Renaming a parameter requires removing all previous
>>>>>>>>>>>       versions that use the old name.
>>>>>>>>>>>    4. We discussed whether parameters or return types can be
>>>>>>>>>>>    nullable and how engines should handle null values. We’ll research SQL
>>>>>>>>>>>    standards and common engine behaviors before finalizing. The current
>>>>>>>>>>>    leaning is:
>>>>>>>>>>>       1. Allow all parameters to be nullable.
>>>>>>>>>>>       2. Allow the return type to include an explicit
>>>>>>>>>>>       nullability annotation.
>>>>>>>>>>>
>>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>>> https://youtu.be/IUd-SN0CQRs
>>>>>>>>>>> Will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>>>>> soon.
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>>>>>>>>>>
>>>>>>>>>>>>    1. Replace “definitions version” with “definitions logs”,
>>>>>>>>>>>>    which record overload changes and timestamps for informational purposes
>>>>>>>>>>>>    only.
>>>>>>>>>>>>    2. Use a globally increasing numeric ID for overloads
>>>>>>>>>>>>    instead of UUIDs.
>>>>>>>>>>>>    3. Add explicit types to all fields in the spec to clarify
>>>>>>>>>>>>    the metadata structure.
>>>>>>>>>>>>    4. Add null-handling optimization hints to help engines
>>>>>>>>>>>>    skip evaluation when inputs are null. (Thanks, Talat!)
>>>>>>>>>>>>    5. Function parameters remain ordered, but we don’t need to
>>>>>>>>>>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>>>>>>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>>>>>>>>    nested structs.
>>>>>>>>>>>>
>>>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>>>> https://youtu.be/gJnueQODWvs
>>>>>>>>>>>> I have updated the PR (
>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) for the above
>>>>>>>>>>>> feedback. Please take a look and let me know if you have any questions.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1. Instead of relying on dynamic inference, the return
>>>>>>>>>>>>>    table’s schema for user-defined table functions (UDTFs) should be
>>>>>>>>>>>>>    explicitly defined.
>>>>>>>>>>>>>    2. Whether a function is a UDTF should be captured as a
>>>>>>>>>>>>>    dedicated attribute, rather than being inferred indirectly from the return
>>>>>>>>>>>>>    type.
>>>>>>>>>>>>>    3. The interpretation of a UDF body (whether it is treated
>>>>>>>>>>>>>    as a partial SQL expression or as a full SELECT statement) should be
>>>>>>>>>>>>>    determined by engines. Example: `SELECT x +1` vs. `x + 1`. Different
>>>>>>>>>>>>>    engines have different takes.
>>>>>>>>>>>>>    4. User-defined aggregation functions (UDAFs) are out of
>>>>>>>>>>>>>    scope for now.
>>>>>>>>>>>>>    5. Each overload should include its own current-version
>>>>>>>>>>>>>    field. This avoids relying solely on the global `definition-versions` when
>>>>>>>>>>>>>    querying the current version of one overload.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>>>>>>>>> I will update the PR(
>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) shortly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Really appreciated feedback from you all over the past few
>>>>>>>>>>>>>> months. I've filed the initial PR for the UDF spec:
>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures
>>>>>>>>>>>>>> the consensus we've built and addresses the write amplification concern
>>>>>>>>>>>>>> raised in our last discussion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please take a look and share your thoughts. Happy to discuss
>>>>>>>>>>>>>> it further during Monday's meeting as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We also discussed a way to avoid copying every overload into
>>>>>>>>>>>>>>> the new metadata JSON when creating a new version. One of ideas is to
>>>>>>>>>>>>>>> introduce a global version array, this is not yet reflected in the doc, but
>>>>>>>>>>>>>>> I’ll update it shortly. Other key points:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>>>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>>>>>>>>>    execute.
>>>>>>>>>>>>>>>    - Keeping the version while referring to an UDF
>>>>>>>>>>>>>>>    probably isn't a good idea. Users are responsible for updating downstream
>>>>>>>>>>>>>>>    views if they reference older UDF versions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In
>>>>>>>>>>>>>>>> general, we discussed the UDF metadata structure, captured at this doc(
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    1. Each UDF overload has its own return type. e.g.,
>>>>>>>>>>>>>>>>    `add(int, int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>>>>>>>>    2. Return type should be explicitly specified, no
>>>>>>>>>>>>>>>>    implicit or statement-based return type inference should be allowed.
>>>>>>>>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>>>>>>>>    properties at the overload level.
>>>>>>>>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>>>>>>>>    5. Introducing a dedicated signature definitions
>>>>>>>>>>>>>>>>    section to centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>>>>>>>>    6. Whether to have versioned open properties or not.
>>>>>>>>>>>>>>>>    Versioned properties can lead to unnecessary copying of a bag of properties
>>>>>>>>>>>>>>>>    into each version, while it provides a clear history of properties for any
>>>>>>>>>>>>>>>>    future debugging and understanding of the UDF behavior at a
>>>>>>>>>>>>>>>>    specific point in time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi everyone, here’s the summary from our last sync on
>>>>>>>>>>>>>>>>> 8/11. Apologies for the delay!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>>>>>>>>       - We agreed to combine overloads with the same name
>>>>>>>>>>>>>>>>>       into a single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>>>>>>>>       - Listing UDFs will return a list of UDF names, not
>>>>>>>>>>>>>>>>>       a list of individual signatures.
>>>>>>>>>>>>>>>>>       - Loading a UDF by name will return all of its
>>>>>>>>>>>>>>>>>       overloads.
>>>>>>>>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>>>>>>>>       - A global version number will track changes across
>>>>>>>>>>>>>>>>>       the entire UDF entity, it increments monolithically.
>>>>>>>>>>>>>>>>>       - Each overload will also maintain its own version
>>>>>>>>>>>>>>>>>       (e.g., updated_at_version) to trace changes specific to that overload.
>>>>>>>>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>>>>>>>>    this stage.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <
>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs
>>>>>>>>>>>>>>>>>> with multiple signatures under the same name, which can serve both
>>>>>>>>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Per my investigation[1], most engines support overloading
>>>>>>>>>>>>>>>>>> by arguments and allow implicit conversions like numeric widening (e.g.,
>>>>>>>>>>>>>>>>>> INT → BIGINT/FLOAT). The resolution approach causes issues like silent
>>>>>>>>>>>>>>>>>> behavior change. Here is an example:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected
>>>>>>>>>>>>>>>>>>    code.
>>>>>>>>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>>>>>>>>    - Engine’s best-match resolution now binds the same
>>>>>>>>>>>>>>>>>>    call to the new overload, changing behavior without modifying the query.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To mitigate this issue, we have to choose between these
>>>>>>>>>>>>>>>>>> two access control models:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all
>>>>>>>>>>>>>>>>>>    overloads of a function name.
>>>>>>>>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to
>>>>>>>>>>>>>>>>>>    specific signatures.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The general recommendation is to adopt *Model A.* It
>>>>>>>>>>>>>>>>>> trades some precision for safety and simplicity, while eliminating the
>>>>>>>>>>>>>>>>>> silent behavior change problem. More details are in this doc[1].
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>> We have discussed how to define function identifiers
>>>>>>>>>>>>>>>>>>> (should also handle function overloading). Ryan suggested that we should
>>>>>>>>>>>>>>>>>>> check how Spark does it. We can refer to functions using an identifier and
>>>>>>>>>>>>>>>>>>> then bind the different signatures to it. So that access policies can be
>>>>>>>>>>>>>>>>>>> applied per identifier. This is also linked to how we want to version the
>>>>>>>>>>>>>>>>>>> functions when overloading is supported.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I will check more about this and update the proposal
>>>>>>>>>>>>>>>>>>> doc.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please check/subscribe to the dev events calendar for
>>>>>>>>>>>>>>>>>>> the next meeting link (Aug 11).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg
>>>>>>>>>>>>>>>>>>>> Dev Events" calendar for tomorrow 7/28 at 9AM PT. I
>>>>>>>>>>>>>>>>>>>> missed the last one, but i'll be at this one.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> No one joined the sync today. I came to know that
>>>>>>>>>>>>>>>>>>>>> Yufei is on holiday, and Ryan and others couldn't make it, similar to the
>>>>>>>>>>>>>>>>>>>>> last sync. It seems Yufei might have forgotten to transfer meeting
>>>>>>>>>>>>>>>>>>>>> ownership as well, as new members needed admin approval and couldn't join
>>>>>>>>>>>>>>>>>>>>> automatically this week. Also, I can understand it is summer holiday season
>>>>>>>>>>>>>>>>>>>>> for many.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I've updated the function signature schema and other
>>>>>>>>>>>>>>>>>>>>> open points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a top
>>>>>>>>>>>>>>>>>>>>>> level field to a field inside “version” along with a representation, so
>>>>>>>>>>>>>>>>>>>>>> that properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> That said, it's still an open question whether we
>>>>>>>>>>>>>>>>>>>>>> need any non-versioned properties. We can introduce them later if a use
>>>>>>>>>>>>>>>>>>>>>> case arises.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from
>>>>>>>>>>>>>>>>>>>>>>> the representation history. Here are reasons:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    1. Each version still enforces a single
>>>>>>>>>>>>>>>>>>>>>>>    signature. Although the signatures array is global to the UDF, each version
>>>>>>>>>>>>>>>>>>>>>>>    references just one signature ID. Rollbacks to historical versions remain
>>>>>>>>>>>>>>>>>>>>>>>    safe.
>>>>>>>>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they
>>>>>>>>>>>>>>>>>>>>>>>    should remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>> We have discussed the action items from the last
>>>>>>>>>>>>>>>>>>>>>>>> sync (*see Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the
>>>>>>>>>>>>>>>>>>>>>>>>    engines and in the roadmaps of many engines. Iceberg will support it. We
>>>>>>>>>>>>>>>>>>>>>>>>    will maintain the `FunctionIdentifier` (extends `TableIdentifer` but also
>>>>>>>>>>>>>>>>>>>>>>>>    have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a
>>>>>>>>>>>>>>>>>>>>>>>>    bag, we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural
>>>>>>>>>>>>>>>>>>>>>>>>    bodies are supported by some engines. Iceberg will support it. Store the
>>>>>>>>>>>>>>>>>>>>>>>>    body as it is while creating function by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    - Standardizing the property names
>>>>>>>>>>>>>>>>>>>>>>>>    (deterministic, secure).
>>>>>>>>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level
>>>>>>>>>>>>>>>>>>>>>>>>    replace is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>>>>>>>>    - Signature should be associated with
>>>>>>>>>>>>>>>>>>>>>>>>    representation?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please review
>>>>>>>>>>>>>>>>>>>>>>>>    the proposal
>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the
>>>>>>>>>>>>>>>>>>>>>>>>> whole metadata is encrypted, we don't have to worry about just hiding the
>>>>>>>>>>>>>>>>>>>>>>>>> UDF body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown
>>>>>>>>>>>>>>>>>>>>>>>>>> are required.We will need a named key(e.g., secure) somewhere, no matter if
>>>>>>>>>>>>>>>>>>>>>>>>>> it is a top level property or a key as a part of the UDF properties. So
>>>>>>>>>>>>>>>>>>>>>>>>>> that both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think
>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec would require? Would it require hiding the UDF definition from
>>>>>>>>>>>>>>>>>>>>>>>>>>> users and require specific pushdown cases be disabled? The use cases seem
>>>>>>>>>>>>>>>>>>>>>>>>>>> valid, but I'm trying to understand the requirements this places on engines
>>>>>>>>>>>>>>>>>>>>>>>>>>> and why it needs to be part of the spec, rather than part of the properties
>>>>>>>>>>>>>>>>>>>>>>>>>>> of the UDF.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    concealing the UDF body and details like the list of imports, some of them
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    aren’t applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    in an isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    For example, secure UDFs may disable certain optimizations—such as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    predicate pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point
>>>>>>>>>>>>>>>>>>>>>>>>>>>> that the secure flag is primarily an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> instruction to the engine to behave differently. While it's largely an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine-side behavior, we still need to include this flag in the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition to indicate whether a UDF is secure, especially considering the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> perf penalty introduced by scenario #3. We should clearly recommend that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> users avoid marking UDFs as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supporting a "secure" UDF? What use case are you addressing and what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specifically changes about how the UDF is handled? If the idea is to hide
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDF definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine". When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but whether those statements run within a single transaction should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> treated as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “secure UDF” option supported by some engines[1], so the body and any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sensitive details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    syntax supported by different engines (Snowflake, databricks, Dremio,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Trino, OSS spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    separator, like $$ or '' or none. Action item was to check whether engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    support multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Need to check whether these engines support function overloading for SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDFs. Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    discussed the deterministic flag and concluded that we keep the independent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    fields (like deterministic) in spec only if the majority of engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    supports it. Else it will be passed in a property bag (engine specific).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    And it is the engine's responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> document here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    the majority agreed *not to* (see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    recording for details).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    will be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    each catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    existing table/view spec — e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    location will remain *required* for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    document
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    vote once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> community sync for the UDF project. Everyone’s welcome to drop in and share
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ideas! Here is the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun.zou to discuss the UDF proposal. We covered several key points, though
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some are still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines may not support vararg syntax in CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION, Iceberg UDFs could represent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently doesn’t support generic types (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> object), we can only map engine-specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Python as a language for SQL UDFs seems promising, especially given its
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> potential to resolve interoperability challenges. Some engines, however,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> require platform version and package dependency details to execute Python
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code—this should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> two primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (open to more interested participants) to continue refining and finalizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> based on the previous comments. Additionally, I've included the SQL UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax supported by various vendors, including Dremio, Snowflake,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a deeper discussion is needed. Let's keep moving forward, especially with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there was significant interest in storing UDFs in Iceberg and adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations to be SQL. It merely does not specify (in this revision)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (adding a new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement of the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concerns as this proposal focuses on SQL-based engines, while Python-based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> systems often work with data frames. Adding imperative languages like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Python would make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs are hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't fully understand yet how the proposed design addresses shareability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal and added comments. I think it would be helpful to also have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> design doc that covers the choices from the draft spec. For instance, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> choice to enumerate all possible function input struts rather than allowing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate function signatures is limiting. It would be nice to see a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion of the trade-offs and a rationale for the choice. I think it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would also be very helpful to have a few representative use cases for this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> included in the doc. That way the proposal can demonstrate that it solves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those use cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> where this is inconsistent with conventions in other specs. For example,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using string IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> model for spec versioning than the Iceberg view and table specs. It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requires readers to fail if there are any unknown fields, which prevents
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec from adding things that are fully backward-compatible. Other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg specs only require a version change to introduce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> forward-incompatible changes and I think that this should do the same to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow multiple function signatures per verison, but it is unclear how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encode them because a version is associated with a single function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax for creating functions across engines, so this doesn’t show that the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function shows a SELECT statement and it isn’t clear how this is distinct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> no more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is an interesting direction, but there might be some details that need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be fine tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> others who might be interested. Resharing since I do not think it was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> directly linked in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> since we didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review so far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attached in the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clarity to take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> meant scalar/aggregate/table user defined functions. Here are some examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referred to is a variation of (1) where the API is data flow/data pipeline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API instead of SQL (e.g., Spark Scala). Yes, that is also possible in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explore some long term opportunities in this case. Consider you register a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark temp view as some sort of data frame read, then it could still be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolved to a Spark plan that is representable by an intermediate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representation. But I agree this gets very complicated very soon, and just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> having the case (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> note that a tabular SQL UDF can be used to build a parameterized view.  So,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there's definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initially supports SQL representations of UDFs (similar to views as shared
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the reference links above), the complexity involved will be similar to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Robert, and Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> publishing the draft spec (inspired by the view spec) this week to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> facilitate further discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different from the views? I feel we can say exactly the same thing for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg views, but yet we have Iceberg multi-dialect views implemented.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe it sounds like we are trying to draw a line between SQL vs other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> programming language as "code"? but I think SQL is just another type of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code, and we are already talking about compiling all these different code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dialects to an intermediate representation (using projects like Coral,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Substrait), which will be stored as another type of representation of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view. I think the same functionality can be used for UDFs if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding UDF support is a good idea, even just a multi-dialect one like view,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and that can allow engines to for example parse a view SQL, and when a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> define a common base upon which engines can build, so the argument that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs aren't practical, because engines are different, is probably only a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temporary concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of adding a negative touch to the idea of having UDFs in Iceberg,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> especially not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about whether it's a good idea to add UDFs tracked by Iceberg catalogs. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think that Iceberg primarily deals with things that are centralized, like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tables of data. While it would be great to have a common set of functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> across engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion to gauge the community interest in storing the Versioned SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> propose the spec addition for storing the versioned UDFs in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operate similarly to views in that they are associated with tables, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they can accept arguments and produce return values, or even function as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines like Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Interoperability between the engines. Potentially engines can understand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDFs written by other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Thanks everyone for joining yesterday's UDF sync. We focused on next steps
for the global parameter name list and on how to approach default values:
1. We made good progress on parameter names by pushing down the parameter
name list to individual overloads and applying constraints on call
conventions.
2. For default values, the current direction is to let engines decide how
they support them, rather than specifying default value semantics in the
UDF spec at this stage.

The meeting recording is available here: https://youtu.be/2uKxYr1yX5w

You can find detailed and corresponding changes in PR
https://github.com/apache/iceberg/pull/14117. The PR is ready for another
look. Feedback is very welcome.

Yufei


On Mon, Jan 12, 2026 at 9:01 AM Yufei Gu <fl...@gmail.com> wrote:

> Hi Folks,
>
> The UDF spec is overall a good sharp. Thanks for all the feedback. In
> today's sync, we could discuss the global parameter name list(drafted one
> google doc for its pros and cons,
> https://docs.google.com/document/d/1ipeEDTlgnBmoH_DJA3HWs6ZiLRaZGSDR-KwS__gc4-M/edit?tab=t.0)
> and whether we need to support the default value at this phase.
>
> Yufei
>
>
> On Mon, Dec 15, 2025 at 2:18 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Thanks everyone for joining today’s UDF sync. We covered several topics:
>>
>> 1. We generally agreed to prefer a simpler form for complex function type
>> signatures, instead of the Iceberg type JSON format (
>> https://iceberg.apache.org/spec/#appendix-c-json-serialization).
>> 2. We discussed the motivations and concerns around having a global
>> parameter name list. Given the trade-offs, we decided to take a deeper look
>> offline.
>> 3. We talked about default values in terms of overload disambiguation and
>> cross-engine support, and agreed that this needs further investigation.
>>
>> Going forward, we’ll use the following Google Doc to record key design
>> decisions and rationale, so we have a shared reference as discussions
>> evolve:
>>
>> https://docs.google.com/document/d/1GKtqFt9gnlJxwo5kfExupDf1jASlgcnDeZv8vEaJQUs/edit?usp=sharing
>>
>> The meeting recording is available here: https://youtu.be/tF7Ur88snR0
>>
>> Thanks Kevin for the proposal. Given the holiday season, we'd like to
>> propose cancelling the next Iceberg UDF Sync scheduled for December 29,
>> 2025. Please let me know if there are any objections.
>>
>> Yufei
>>
>>
>> On Fri, Dec 5, 2025 at 8:51 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Thanks for joining Monday’s UDF sync!
>>> We covered two updates:
>>>
>>>    - Parameter names moved to the top level, this keeps named-argument
>>>    invocation consistent across overloads.
>>>    - Default value support added, callers can now omit parameters that
>>>    have defaults.
>>>
>>> The recording is available here for more details:
>>> https://youtu.be/6CBMwwbISOE
>>>
>>> I’ve also added more examples demonstrating default values. Please take
>>> a look at the PR: https://github.com/apache/iceberg/pull/14117
>>>
>>>
>>> Yufei
>>>
>>>
>>> On Wed, Nov 26, 2025 at 5:57 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Made a few changes to the spec. The PR(
>>>> https://github.com/apache/iceberg/pull/14117) is ready for another
>>>> look.
>>>> 1. Moved the parameter name list to the top level, so that
>>>> named-argument invocation is consistent across overloads.
>>>> 2. Added support for default value, which allows caller to ignore
>>>> parameters with default values.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Nov 17, 2025 at 6:42 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap of
>>>>> what we discussed and agreed on:
>>>>>
>>>>>    1. Huaxin walked us through the Spark POC(
>>>>>    https://github.com/apache/iceberg/pull/14505). Thanks Huaxin!
>>>>>    2. When default values are enabled, ambiguous overloads must not
>>>>>    be allowed. We agreed that definition-id alone is insufficient for conflict
>>>>>    detection; additional signature-level check across overloads is needed.
>>>>>    3. We will explore the idea that parameter names are globally
>>>>>    consistent across multiple overloads. For example, if the overload foo(int
>>>>>    x, int y) exists, foo(int a) is not allowed, but foo(int x) is OK. We could
>>>>>    put a list of names and their docs at the top level. This benefits UX and
>>>>>    may potentially benefit the strategy to pick up the overload to use given
>>>>>    the inputs. We may bring back the parameter names at the representation
>>>>>    level, so that it's more flexible.
>>>>>    4. Should an engine ignore the default values or fail fast if it
>>>>>    doesn’t support it? We concluded that this is up to the engines, and
>>>>>    engines could enforce all parameters to be provided when default values
>>>>>    aren't supported.
>>>>>    5. How do we give the default value for complex types(e.g.
>>>>>    struct)? We should follow the way table spec handles the default value(
>>>>>    https://iceberg.apache.org/spec/#edge-interpolation-algorithm).
>>>>>
>>>>> You can watch the recording here for more details:
>>>>> https://youtu.be/Wxz1nSP48k0
>>>>>
>>>>> Please take another look at PR
>>>>> https://github.com/apache/iceberg/pull/14117 when you get a chance.
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Fri, Nov 14, 2025 at 4:29 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> The PR (https://github.com/apache/iceberg/pull/14117) has been
>>>>>> updated, please take another look.
>>>>>> For next Monday’s UDF sync, Huaxin will demo a Spark POC showing how
>>>>>> the current (WIP) UDF spec works. We can also walk through how default
>>>>>> values should be introduced into the spec.
>>>>>>
>>>>>> *TL;DR:* I haven’t added default-value support in the PR yet. It’s
>>>>>> turning out to be trickier than expected because default values make
>>>>>> parameters optional. That introduces ambiguity for overloads like:
>>>>>>
>>>>>>    -
>>>>>>
>>>>>>    foo(int)
>>>>>>    -
>>>>>>
>>>>>>    foo(int, int default 0)
>>>>>>    -
>>>>>>
>>>>>>    foo(int, float default 1.0)
>>>>>>
>>>>>> All of these could be invoked as foo(int). The first question is:
>>>>>> should we allow such overloads to coexist? Probably not, both Postgres and
>>>>>> Snowflake support default values, but neither allows this form of ambiguous
>>>>>> overloading.
>>>>>> Assuming we disallow it, the current UDF definition-id (type tuple)
>>>>>> isn’t sufficient to detect conflicts. We also need to check the signatures
>>>>>> of existing overloads to determine whether a new one is valid. For example,
>>>>>> if (int, int) exists, then neither (int) nor (int, float) should be
>>>>>> allowed.
>>>>>>
>>>>>> Reference:
>>>>>> Postgres behavior on default parameters:
>>>>>> https://www.postgresql.org/docs/current/static/typeconv-func.html
>>>>>> Functions with default arguments match calls that omit zero or more
>>>>>> defaultable parameters. If multiple functions match, the earliest in the
>>>>>> search path is chosen; if they are equally specific, Postgres throws an
>>>>>> “ambiguous function call” error.
>>>>>> Thanks,
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap
>>>>>>> of what we discussed and agreed on:
>>>>>>> 1. We’ll keep parameters only at the definition layer to support
>>>>>>> named-argument invocation without extra work from the engine side.
>>>>>>> 2. We agreed to add support for default parameter values.
>>>>>>> 3. Moving forward with a flag for nullable vs. non-nullable return
>>>>>>> values, treating it as a hint rather than a strict rule.
>>>>>>> 4. The udf metadata will preserve the original case of parameter
>>>>>>> names, and engines can decide whether to normalize or convert them.
>>>>>>>
>>>>>>> You can watch the recording here for more details:
>>>>>>> https://www.youtube.com/watch?v=0i6nKfDNdGw
>>>>>>> Sorry that today's sync wasn't in the Iceberg Dev Event calendar(It
>>>>>>> was removed somehow). I have added it back to Iceberg Dev Event as a
>>>>>>> biweekly sync. Let me know if you have any questions.
>>>>>>>
>>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>> for item 1 and 2 soon. The overall content will remain largely the same.
>>>>>>> Please take a look when you get a chance!
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Ryan and me are in this meeting, please join
>>>>>>>>
>>>>>>>> Iceberg UDF sync
>>>>>>>> Monday, November 3 · 9:00 – 10:00am
>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>> Google Meet joining info
>>>>>>>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> Thanks a lot for the review! Made a few changes to the spec:
>>>>>>>>> 1. Renamed "overload" to "definition"
>>>>>>>>> 2. Changed definition id from long to string with the format of
>>>>>>>>> parameter type tuple.
>>>>>>>>> 3. Explicitly claimed that named argument invocation is supported
>>>>>>>>> 4. Added a nullable flag for return values
>>>>>>>>> 5. Added a few rules for inputs and returns
>>>>>>>>>
>>>>>>>>> Please find more details in the PR(
>>>>>>>>> https://github.com/apache/iceberg/pull/14117). Here is a doc
>>>>>>>>> comparing engine behaviors in terms of named argument invocation, nullable
>>>>>>>>> return value, default values for inputs, null handling:
>>>>>>>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>>>>>>>> This can help for decision making.
>>>>>>>>>
>>>>>>>>> We could discuss the following items in Monday's meeting:
>>>>>>>>> 1. Whether we need to push down parameters to representation given
>>>>>>>>> we want to support named argument invocation, it requires extra mapping on
>>>>>>>>> the engine side.
>>>>>>>>> 2. Whether we support default values.  Spark plans to support it,
>>>>>>>>> Snowflake supports it already.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> Thanks for joining today’s community UDF sync. Here’s a summary
>>>>>>>>>> of what we discussed and agreed upon:
>>>>>>>>>>
>>>>>>>>>>    1. We agreed to use Iceberg types for each field in the UDF
>>>>>>>>>>    spec. This approach ensures consistency with the table spec and allows for
>>>>>>>>>>    straightforward conversion to JSON types.
>>>>>>>>>>    2. Using numeric overload IDs can have side effects, for
>>>>>>>>>>    instance, deleting an overload and later creating a new one with the same
>>>>>>>>>>    signature will use a different numeric id, which points to the same
>>>>>>>>>>    overload. The consensus is to represent the overload ID as a string derived
>>>>>>>>>>    from parameter types and their order.
>>>>>>>>>>    3. Parameter names must remain consistent across all overload
>>>>>>>>>>    versions. We agreed to add explicit rules:
>>>>>>>>>>       1. Parameter names must match across all overload versions.
>>>>>>>>>>       2. Renaming a parameter requires removing all previous
>>>>>>>>>>       versions that use the old name.
>>>>>>>>>>    4. We discussed whether parameters or return types can be
>>>>>>>>>>    nullable and how engines should handle null values. We’ll research SQL
>>>>>>>>>>    standards and common engine behaviors before finalizing. The current
>>>>>>>>>>    leaning is:
>>>>>>>>>>       1. Allow all parameters to be nullable.
>>>>>>>>>>       2. Allow the return type to include an explicit
>>>>>>>>>>       nullability annotation.
>>>>>>>>>>
>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>> https://youtu.be/IUd-SN0CQRs
>>>>>>>>>> Will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>>>> soon.
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>>>>>>>>>
>>>>>>>>>>>    1. Replace “definitions version” with “definitions logs”,
>>>>>>>>>>>    which record overload changes and timestamps for informational purposes
>>>>>>>>>>>    only.
>>>>>>>>>>>    2. Use a globally increasing numeric ID for overloads
>>>>>>>>>>>    instead of UUIDs.
>>>>>>>>>>>    3. Add explicit types to all fields in the spec to clarify
>>>>>>>>>>>    the metadata structure.
>>>>>>>>>>>    4. Add null-handling optimization hints to help engines skip
>>>>>>>>>>>    evaluation when inputs are null. (Thanks, Talat!)
>>>>>>>>>>>    5. Function parameters remain ordered, but we don’t need to
>>>>>>>>>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>>>>>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>>>>>>>    nested structs.
>>>>>>>>>>>
>>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>>> https://youtu.be/gJnueQODWvs
>>>>>>>>>>> I have updated the PR (
>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) for the above
>>>>>>>>>>> feedback. Please take a look and let me know if you have any questions.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>>>>>>>>>
>>>>>>>>>>>>    1. Instead of relying on dynamic inference, the return
>>>>>>>>>>>>    table’s schema for user-defined table functions (UDTFs) should be
>>>>>>>>>>>>    explicitly defined.
>>>>>>>>>>>>    2. Whether a function is a UDTF should be captured as a
>>>>>>>>>>>>    dedicated attribute, rather than being inferred indirectly from the return
>>>>>>>>>>>>    type.
>>>>>>>>>>>>    3. The interpretation of a UDF body (whether it is treated
>>>>>>>>>>>>    as a partial SQL expression or as a full SELECT statement) should be
>>>>>>>>>>>>    determined by engines. Example: `SELECT x +1` vs. `x + 1`. Different
>>>>>>>>>>>>    engines have different takes.
>>>>>>>>>>>>    4. User-defined aggregation functions (UDAFs) are out of
>>>>>>>>>>>>    scope for now.
>>>>>>>>>>>>    5. Each overload should include its own current-version
>>>>>>>>>>>>    field. This avoids relying solely on the global `definition-versions` when
>>>>>>>>>>>>    querying the current version of one overload.
>>>>>>>>>>>>
>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>>>>>>>> I will update the PR(
>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) shortly.
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Really appreciated feedback from you all over the past few
>>>>>>>>>>>>> months. I've filed the initial PR for the UDF spec:
>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>>>>>>>>>>> consensus we've built and addresses the write amplification concern raised
>>>>>>>>>>>>> in our last discussion.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please take a look and share your thoughts. Happy to discuss
>>>>>>>>>>>>> it further during Monday's meeting as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We also discussed a way to avoid copying every overload into
>>>>>>>>>>>>>> the new metadata JSON when creating a new version. One of ideas is to
>>>>>>>>>>>>>> introduce a global version array, this is not yet reflected in the doc, but
>>>>>>>>>>>>>> I’ll update it shortly. Other key points:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>>>>>>>>    execute.
>>>>>>>>>>>>>>    - Keeping the version while referring to an UDF
>>>>>>>>>>>>>>    probably isn't a good idea. Users are responsible for updating downstream
>>>>>>>>>>>>>>    views if they reference older UDF versions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <
>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In general,
>>>>>>>>>>>>>>> we discussed the UDF metadata structure, captured at this doc(
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    1. Each UDF overload has its own return type. e.g.,
>>>>>>>>>>>>>>>    `add(int, int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>>>>>>>    2. Return type should be explicitly specified, no
>>>>>>>>>>>>>>>    implicit or statement-based return type inference should be allowed.
>>>>>>>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>>>>>>>    properties at the overload level.
>>>>>>>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>>>>>>>    5. Introducing a dedicated signature definitions section
>>>>>>>>>>>>>>>    to centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>>>>>>>    6. Whether to have versioned open properties or not.
>>>>>>>>>>>>>>>    Versioned properties can lead to unnecessary copying of a bag of properties
>>>>>>>>>>>>>>>    into each version, while it provides a clear history of properties for any
>>>>>>>>>>>>>>>    future debugging and understanding of the UDF behavior at a
>>>>>>>>>>>>>>>    specific point in time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>>>>>>>>>>> Apologies for the delay!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>>>>>>>       - We agreed to combine overloads with the same name
>>>>>>>>>>>>>>>>       into a single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>>>>>>>       - Listing UDFs will return a list of UDF names, not
>>>>>>>>>>>>>>>>       a list of individual signatures.
>>>>>>>>>>>>>>>>       - Loading a UDF by name will return all of its
>>>>>>>>>>>>>>>>       overloads.
>>>>>>>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>>>>>>>       - A global version number will track changes across
>>>>>>>>>>>>>>>>       the entire UDF entity, it increments monolithically.
>>>>>>>>>>>>>>>>       - Each overload will also maintain its own version
>>>>>>>>>>>>>>>>       (e.g., updated_at_version) to trace changes specific to that overload.
>>>>>>>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>>>>>>>    this stage.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs with
>>>>>>>>>>>>>>>>> multiple signatures under the same name, which can serve both
>>>>>>>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Per my investigation[1], most engines support overloading
>>>>>>>>>>>>>>>>> by arguments and allow implicit conversions like numeric widening (e.g.,
>>>>>>>>>>>>>>>>> INT → BIGINT/FLOAT). The resolution approach causes issues like silent
>>>>>>>>>>>>>>>>> behavior change. Here is an example:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected
>>>>>>>>>>>>>>>>>    code.
>>>>>>>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>>>>>>>    - Engine’s best-match resolution now binds the same
>>>>>>>>>>>>>>>>>    call to the new overload, changing behavior without modifying the query.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To mitigate this issue, we have to choose between these
>>>>>>>>>>>>>>>>> two access control models:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all
>>>>>>>>>>>>>>>>>    overloads of a function name.
>>>>>>>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to
>>>>>>>>>>>>>>>>>    specific signatures.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The general recommendation is to adopt *Model A.* It
>>>>>>>>>>>>>>>>> trades some precision for safety and simplicity, while eliminating the
>>>>>>>>>>>>>>>>> silent behavior change problem. More details are in this doc[1].
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>> We have discussed how to define function identifiers
>>>>>>>>>>>>>>>>>> (should also handle function overloading). Ryan suggested that we should
>>>>>>>>>>>>>>>>>> check how Spark does it. We can refer to functions using an identifier and
>>>>>>>>>>>>>>>>>> then bind the different signatures to it. So that access policies can be
>>>>>>>>>>>>>>>>>> applied per identifier. This is also linked to how we want to version the
>>>>>>>>>>>>>>>>>> functions when overloading is supported.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please check/subscribe to the dev events calendar for the
>>>>>>>>>>>>>>>>>> next meeting link (Aug 11).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg
>>>>>>>>>>>>>>>>>>> Dev Events" calendar for tomorrow 7/28 at 9AM PT. I
>>>>>>>>>>>>>>>>>>> missed the last one, but i'll be at this one.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> No one joined the sync today. I came to know that Yufei
>>>>>>>>>>>>>>>>>>>> is on holiday, and Ryan and others couldn't make it, similar to the last
>>>>>>>>>>>>>>>>>>>> sync. It seems Yufei might have forgotten to transfer meeting ownership as
>>>>>>>>>>>>>>>>>>>> well, as new members needed admin approval and couldn't join automatically
>>>>>>>>>>>>>>>>>>>> this week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've updated the function signature schema and other
>>>>>>>>>>>>>>>>>>>> open points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a top
>>>>>>>>>>>>>>>>>>>>> level field to a field inside “version” along with a representation, so
>>>>>>>>>>>>>>>>>>>>> that properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> That said, it's still an open question whether we need
>>>>>>>>>>>>>>>>>>>>> any non-versioned properties. We can introduce them later if a use case
>>>>>>>>>>>>>>>>>>>>> arises.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from
>>>>>>>>>>>>>>>>>>>>>> the representation history. Here are reasons:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    1. Each version still enforces a single
>>>>>>>>>>>>>>>>>>>>>>    signature. Although the signatures array is global to the UDF, each version
>>>>>>>>>>>>>>>>>>>>>>    references just one signature ID. Rollbacks to historical versions remain
>>>>>>>>>>>>>>>>>>>>>>    safe.
>>>>>>>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they
>>>>>>>>>>>>>>>>>>>>>>    should remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>> We have discussed the action items from the last
>>>>>>>>>>>>>>>>>>>>>>> sync (*see Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the
>>>>>>>>>>>>>>>>>>>>>>>    engines and in the roadmaps of many engines. Iceberg will support it. We
>>>>>>>>>>>>>>>>>>>>>>>    will maintain the `FunctionIdentifier` (extends `TableIdentifer` but also
>>>>>>>>>>>>>>>>>>>>>>>    have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a
>>>>>>>>>>>>>>>>>>>>>>>    bag, we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies
>>>>>>>>>>>>>>>>>>>>>>>    are supported by some engines. Iceberg will support it. Store the body as
>>>>>>>>>>>>>>>>>>>>>>>    it is while creating function by the engine.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    - Standardizing the property names
>>>>>>>>>>>>>>>>>>>>>>>    (deterministic, secure).
>>>>>>>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level
>>>>>>>>>>>>>>>>>>>>>>>    replace is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>>>>>>>    - Signature should be associated with
>>>>>>>>>>>>>>>>>>>>>>>    representation?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please review
>>>>>>>>>>>>>>>>>>>>>>>    the proposal
>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the
>>>>>>>>>>>>>>>>>>>>>>>> whole metadata is encrypted, we don't have to worry about just hiding the
>>>>>>>>>>>>>>>>>>>>>>>> UDF body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown
>>>>>>>>>>>>>>>>>>>>>>>>> are required.We will need a named key(e.g., secure) somewhere, no matter if
>>>>>>>>>>>>>>>>>>>>>>>>> it is a top level property or a key as a part of the UDF properties. So
>>>>>>>>>>>>>>>>>>>>>>>>> that both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think
>>>>>>>>>>>>>>>>>>>>>>>>>> the spec would require? Would it require hiding the UDF definition from
>>>>>>>>>>>>>>>>>>>>>>>>>> users and require specific pushdown cases be disabled? The use cases seem
>>>>>>>>>>>>>>>>>>>>>>>>>> valid, but I'm trying to understand the requirements this places on engines
>>>>>>>>>>>>>>>>>>>>>>>>>> and why it needs to be part of the spec, rather than part of the properties
>>>>>>>>>>>>>>>>>>>>>>>>>> of the UDF.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes
>>>>>>>>>>>>>>>>>>>>>>>>>>>    concealing the UDF body and details like the list of imports, some of them
>>>>>>>>>>>>>>>>>>>>>>>>>>>    aren’t applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs
>>>>>>>>>>>>>>>>>>>>>>>>>>>    in an isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time:
>>>>>>>>>>>>>>>>>>>>>>>>>>>    For example, secure UDFs may disable certain optimizations—such as
>>>>>>>>>>>>>>>>>>>>>>>>>>>    predicate pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point
>>>>>>>>>>>>>>>>>>>>>>>>>>> that the secure flag is primarily an
>>>>>>>>>>>>>>>>>>>>>>>>>>> instruction to the engine to behave differently. While it's largely an
>>>>>>>>>>>>>>>>>>>>>>>>>>> engine-side behavior, we still need to include this flag in the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>> definition to indicate whether a UDF is secure, especially considering the
>>>>>>>>>>>>>>>>>>>>>>>>>>> perf penalty introduced by scenario #3. We should clearly recommend that
>>>>>>>>>>>>>>>>>>>>>>>>>>> users avoid marking UDFs as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> supporting a "secure" UDF? What use case are you addressing and what
>>>>>>>>>>>>>>>>>>>>>>>>>>>> specifically changes about how the UDF is handled? If the idea is to hide
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDF definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted
>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine". When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but whether those statements run within a single transaction should be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> treated as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “secure UDF” option supported by some engines[1], so the body and any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sensitive details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    separator, like $$ or '' or none. Action item was to check whether engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    support multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Need to check whether these engines support function overloading for SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDFs. Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    discussed the deterministic flag and concluded that we keep the independent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    fields (like deterministic) in spec only if the majority of engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    supports it. Else it will be passed in a property bag (engine specific).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    And it is the engine's responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> document here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    the majority agreed *not to* (see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    recording for details).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    will be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    each catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    existing table/view spec — e.g., location
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    will remain *required* for cross-catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    vote once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> community sync for the UDF project. Everyone’s welcome to drop in and share
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ideas! Here is the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun.zou to discuss the UDF proposal. We covered several key points, though
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some are still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines may not support vararg syntax in CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION, Iceberg UDFs could represent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently doesn’t support generic types (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> object), we can only map engine-specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as a language for SQL UDFs seems promising, especially given its potential
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> two primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to more interested participants) to continue refining and finalizing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the previous comments. Additionally, I've included the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported by various vendors, including Dremio, Snowflake, Databricks, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a deeper discussion is needed. Let's keep moving forward, especially with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there was significant interest in storing UDFs in Iceberg and adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations to be SQL. It merely does not specify (in this revision)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (adding a new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement of the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concerns as this proposal focuses on SQL-based engines, while Python-based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> systems often work with data frames. Adding imperative languages like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Python would make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs are hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't fully understand yet how the proposed design addresses shareability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal and added comments. I think it would be helpful to also have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> design doc that covers the choices from the draft spec. For instance, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> choice to enumerate all possible function input struts rather than allowing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate function signatures is limiting. It would be nice to see a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion of the trade-offs and a rationale for the choice. I think it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would also be very helpful to have a few representative use cases for this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> included in the doc. That way the proposal can demonstrate that it solves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those use cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is inconsistent with conventions in other specs. For example, using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> string IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> model for spec versioning than the Iceberg view and table specs. It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requires readers to fail if there are any unknown fields, which prevents
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec from adding things that are fully backward-compatible. Other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg specs only require a version change to introduce
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> forward-incompatible changes and I think that this should do the same to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow multiple function signatures per verison, but it is unclear how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encode them because a version is associated with a single function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax for creating functions across engines, so this doesn’t show that the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function shows a SELECT statement and it isn’t clear how this is distinct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> no more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is an interesting direction, but there might be some details that need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be fine tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> others who might be interested. Resharing since I do not think it was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> directly linked in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> since we didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review so far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> attached in the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> meant scalar/aggregate/table user defined functions. Here are some examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referred to is a variation of (1) where the API is data flow/data pipeline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API instead of SQL (e.g., Spark Scala). Yes, that is also possible in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explore some long term opportunities in this case. Consider you register a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark temp view as some sort of data frame read, then it could still be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolved to a Spark plan that is representable by an intermediate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representation. But I agree this gets very complicated very soon, and just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> having the case (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> note that a tabular SQL UDF can be used to build a parameterized view.  So,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there's definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initially supports SQL representations of UDFs (similar to views as shared
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the reference links above), the complexity involved will be similar to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> publishing the draft spec (inspired by the view spec) this week to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> facilitate further discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different from the views? I feel we can say exactly the same thing for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg views, but yet we have Iceberg multi-dialect views implemented.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe it sounds like we are trying to draw a line between SQL vs other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> programming language as "code"? but I think SQL is just another type of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code, and we are already talking about compiling all these different code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dialects to an intermediate representation (using projects like Coral,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Substrait), which will be stored as another type of representation of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg view. I think the same functionality can be used for UDFs if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding UDF support is a good idea, even just a multi-dialect one like view,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and that can allow engines to for example parse a view SQL, and when a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 1:32 AM Robert Stupp <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> define a common base upon which engines can build, so the argument that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs aren't practical, because engines are different, is probably only a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temporary concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of adding a negative touch to the idea of having UDFs in Iceberg,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> especially not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether it's a good idea to add UDFs tracked by Iceberg catalogs. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that Iceberg primarily deals with things that are centralized, like tables
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion to gauge the community interest in storing the Versioned SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec addition for storing the versioned UDFs in Iceberg (inspired by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operate similarly to views in that they are associated with tables, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they can accept arguments and produce return values, or even function as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines. Potentially engines can understand the UDFs written by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi Folks,

The UDF spec is overall a good sharp. Thanks for all the feedback. In
today's sync, we could discuss the global parameter name list(drafted one
google doc for its pros and cons,
https://docs.google.com/document/d/1ipeEDTlgnBmoH_DJA3HWs6ZiLRaZGSDR-KwS__gc4-M/edit?tab=t.0)
and whether we need to support the default value at this phase.

Yufei


On Mon, Dec 15, 2025 at 2:18 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks everyone for joining today’s UDF sync. We covered several topics:
>
> 1. We generally agreed to prefer a simpler form for complex function type
> signatures, instead of the Iceberg type JSON format (
> https://iceberg.apache.org/spec/#appendix-c-json-serialization).
> 2. We discussed the motivations and concerns around having a global
> parameter name list. Given the trade-offs, we decided to take a deeper look
> offline.
> 3. We talked about default values in terms of overload disambiguation and
> cross-engine support, and agreed that this needs further investigation.
>
> Going forward, we’ll use the following Google Doc to record key design
> decisions and rationale, so we have a shared reference as discussions
> evolve:
>
> https://docs.google.com/document/d/1GKtqFt9gnlJxwo5kfExupDf1jASlgcnDeZv8vEaJQUs/edit?usp=sharing
>
> The meeting recording is available here: https://youtu.be/tF7Ur88snR0
>
> Thanks Kevin for the proposal. Given the holiday season, we'd like to
> propose cancelling the next Iceberg UDF Sync scheduled for December 29,
> 2025. Please let me know if there are any objections.
>
> Yufei
>
>
> On Fri, Dec 5, 2025 at 8:51 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks for joining Monday’s UDF sync!
>> We covered two updates:
>>
>>    - Parameter names moved to the top level, this keeps named-argument
>>    invocation consistent across overloads.
>>    - Default value support added, callers can now omit parameters that
>>    have defaults.
>>
>> The recording is available here for more details:
>> https://youtu.be/6CBMwwbISOE
>>
>> I’ve also added more examples demonstrating default values. Please take a
>> look at the PR: https://github.com/apache/iceberg/pull/14117
>>
>>
>> Yufei
>>
>>
>> On Wed, Nov 26, 2025 at 5:57 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Made a few changes to the spec. The PR(
>>> https://github.com/apache/iceberg/pull/14117) is ready for another
>>> look.
>>> 1. Moved the parameter name list to the top level, so that
>>> named-argument invocation is consistent across overloads.
>>> 2. Added support for default value, which allows caller to ignore
>>> parameters with default values.
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Nov 17, 2025 at 6:42 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap of
>>>> what we discussed and agreed on:
>>>>
>>>>    1. Huaxin walked us through the Spark POC(
>>>>    https://github.com/apache/iceberg/pull/14505). Thanks Huaxin!
>>>>    2. When default values are enabled, ambiguous overloads must not be
>>>>    allowed. We agreed that definition-id alone is insufficient for conflict
>>>>    detection; additional signature-level check across overloads is needed.
>>>>    3. We will explore the idea that parameter names are globally
>>>>    consistent across multiple overloads. For example, if the overload foo(int
>>>>    x, int y) exists, foo(int a) is not allowed, but foo(int x) is OK. We could
>>>>    put a list of names and their docs at the top level. This benefits UX and
>>>>    may potentially benefit the strategy to pick up the overload to use given
>>>>    the inputs. We may bring back the parameter names at the representation
>>>>    level, so that it's more flexible.
>>>>    4. Should an engine ignore the default values or fail fast if it
>>>>    doesn’t support it? We concluded that this is up to the engines, and
>>>>    engines could enforce all parameters to be provided when default values
>>>>    aren't supported.
>>>>    5. How do we give the default value for complex types(e.g. struct)?
>>>>    We should follow the way table spec handles the default value(
>>>>    https://iceberg.apache.org/spec/#edge-interpolation-algorithm).
>>>>
>>>> You can watch the recording here for more details:
>>>> https://youtu.be/Wxz1nSP48k0
>>>>
>>>> Please take another look at PR
>>>> https://github.com/apache/iceberg/pull/14117 when you get a chance.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Fri, Nov 14, 2025 at 4:29 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> The PR (https://github.com/apache/iceberg/pull/14117) has been
>>>>> updated, please take another look.
>>>>> For next Monday’s UDF sync, Huaxin will demo a Spark POC showing how
>>>>> the current (WIP) UDF spec works. We can also walk through how default
>>>>> values should be introduced into the spec.
>>>>>
>>>>> *TL;DR:* I haven’t added default-value support in the PR yet. It’s
>>>>> turning out to be trickier than expected because default values make
>>>>> parameters optional. That introduces ambiguity for overloads like:
>>>>>
>>>>>    -
>>>>>
>>>>>    foo(int)
>>>>>    -
>>>>>
>>>>>    foo(int, int default 0)
>>>>>    -
>>>>>
>>>>>    foo(int, float default 1.0)
>>>>>
>>>>> All of these could be invoked as foo(int). The first question is:
>>>>> should we allow such overloads to coexist? Probably not, both Postgres and
>>>>> Snowflake support default values, but neither allows this form of ambiguous
>>>>> overloading.
>>>>> Assuming we disallow it, the current UDF definition-id (type tuple)
>>>>> isn’t sufficient to detect conflicts. We also need to check the signatures
>>>>> of existing overloads to determine whether a new one is valid. For example,
>>>>> if (int, int) exists, then neither (int) nor (int, float) should be
>>>>> allowed.
>>>>>
>>>>> Reference:
>>>>> Postgres behavior on default parameters:
>>>>> https://www.postgresql.org/docs/current/static/typeconv-func.html
>>>>> Functions with default arguments match calls that omit zero or more
>>>>> defaultable parameters. If multiple functions match, the earliest in the
>>>>> search path is chosen; if they are equally specific, Postgres throws an
>>>>> “ambiguous function call” error.
>>>>> Thanks,
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap
>>>>>> of what we discussed and agreed on:
>>>>>> 1. We’ll keep parameters only at the definition layer to support
>>>>>> named-argument invocation without extra work from the engine side.
>>>>>> 2. We agreed to add support for default parameter values.
>>>>>> 3. Moving forward with a flag for nullable vs. non-nullable return
>>>>>> values, treating it as a hint rather than a strict rule.
>>>>>> 4. The udf metadata will preserve the original case of parameter
>>>>>> names, and engines can decide whether to normalize or convert them.
>>>>>>
>>>>>> You can watch the recording here for more details:
>>>>>> https://www.youtube.com/watch?v=0i6nKfDNdGw
>>>>>> Sorry that today's sync wasn't in the Iceberg Dev Event calendar(It
>>>>>> was removed somehow). I have added it back to Iceberg Dev Event as a
>>>>>> biweekly sync. Let me know if you have any questions.
>>>>>>
>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>> for item 1 and 2 soon. The overall content will remain largely the same.
>>>>>> Please take a look when you get a chance!
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>>>
>>>>>>> Ryan and me are in this meeting, please join
>>>>>>>
>>>>>>> Iceberg UDF sync
>>>>>>> Monday, November 3 · 9:00 – 10:00am
>>>>>>> Time zone: America/Los_Angeles
>>>>>>> Google Meet joining info
>>>>>>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> Thanks a lot for the review! Made a few changes to the spec:
>>>>>>>> 1. Renamed "overload" to "definition"
>>>>>>>> 2. Changed definition id from long to string with the format of
>>>>>>>> parameter type tuple.
>>>>>>>> 3. Explicitly claimed that named argument invocation is supported
>>>>>>>> 4. Added a nullable flag for return values
>>>>>>>> 5. Added a few rules for inputs and returns
>>>>>>>>
>>>>>>>> Please find more details in the PR(
>>>>>>>> https://github.com/apache/iceberg/pull/14117). Here is a doc
>>>>>>>> comparing engine behaviors in terms of named argument invocation, nullable
>>>>>>>> return value, default values for inputs, null handling:
>>>>>>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>>>>>>> This can help for decision making.
>>>>>>>>
>>>>>>>> We could discuss the following items in Monday's meeting:
>>>>>>>> 1. Whether we need to push down parameters to representation given
>>>>>>>> we want to support named argument invocation, it requires extra mapping on
>>>>>>>> the engine side.
>>>>>>>> 2. Whether we support default values.  Spark plans to support it,
>>>>>>>> Snowflake supports it already.
>>>>>>>>
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> Thanks for joining today’s community UDF sync. Here’s a summary of
>>>>>>>>> what we discussed and agreed upon:
>>>>>>>>>
>>>>>>>>>    1. We agreed to use Iceberg types for each field in the UDF
>>>>>>>>>    spec. This approach ensures consistency with the table spec and allows for
>>>>>>>>>    straightforward conversion to JSON types.
>>>>>>>>>    2. Using numeric overload IDs can have side effects, for
>>>>>>>>>    instance, deleting an overload and later creating a new one with the same
>>>>>>>>>    signature will use a different numeric id, which points to the same
>>>>>>>>>    overload. The consensus is to represent the overload ID as a string derived
>>>>>>>>>    from parameter types and their order.
>>>>>>>>>    3. Parameter names must remain consistent across all overload
>>>>>>>>>    versions. We agreed to add explicit rules:
>>>>>>>>>       1. Parameter names must match across all overload versions.
>>>>>>>>>       2. Renaming a parameter requires removing all previous
>>>>>>>>>       versions that use the old name.
>>>>>>>>>    4. We discussed whether parameters or return types can be
>>>>>>>>>    nullable and how engines should handle null values. We’ll research SQL
>>>>>>>>>    standards and common engine behaviors before finalizing. The current
>>>>>>>>>    leaning is:
>>>>>>>>>       1. Allow all parameters to be nullable.
>>>>>>>>>       2. Allow the return type to include an explicit nullability
>>>>>>>>>       annotation.
>>>>>>>>>
>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>> https://youtu.be/IUd-SN0CQRs
>>>>>>>>> Will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>>> soon.
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>>>>>>>>
>>>>>>>>>>    1. Replace “definitions version” with “definitions logs”,
>>>>>>>>>>    which record overload changes and timestamps for informational purposes
>>>>>>>>>>    only.
>>>>>>>>>>    2. Use a globally increasing numeric ID for overloads instead
>>>>>>>>>>    of UUIDs.
>>>>>>>>>>    3. Add explicit types to all fields in the spec to clarify
>>>>>>>>>>    the metadata structure.
>>>>>>>>>>    4. Add null-handling optimization hints to help engines skip
>>>>>>>>>>    evaluation when inputs are null. (Thanks, Talat!)
>>>>>>>>>>    5. Function parameters remain ordered, but we don’t need to
>>>>>>>>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>>>>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>>>>>>    nested structs.
>>>>>>>>>>
>>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>>> https://youtu.be/gJnueQODWvs
>>>>>>>>>> I have updated the PR (
>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) for the above
>>>>>>>>>> feedback. Please take a look and let me know if you have any questions.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi folks,
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>>>>>>>>
>>>>>>>>>>>    1. Instead of relying on dynamic inference, the return
>>>>>>>>>>>    table’s schema for user-defined table functions (UDTFs) should be
>>>>>>>>>>>    explicitly defined.
>>>>>>>>>>>    2. Whether a function is a UDTF should be captured as a
>>>>>>>>>>>    dedicated attribute, rather than being inferred indirectly from the return
>>>>>>>>>>>    type.
>>>>>>>>>>>    3. The interpretation of a UDF body (whether it is treated
>>>>>>>>>>>    as a partial SQL expression or as a full SELECT statement) should be
>>>>>>>>>>>    determined by engines. Example: `SELECT x +1` vs. `x + 1`. Different
>>>>>>>>>>>    engines have different takes.
>>>>>>>>>>>    4. User-defined aggregation functions (UDAFs) are out of
>>>>>>>>>>>    scope for now.
>>>>>>>>>>>    5. Each overload should include its own current-version
>>>>>>>>>>>    field. This avoids relying solely on the global `definition-versions` when
>>>>>>>>>>>    querying the current version of one overload.
>>>>>>>>>>>
>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>>>>>>> I will update the PR(
>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117) shortly.
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>
>>>>>>>>>>>> Really appreciated feedback from you all over the past few
>>>>>>>>>>>> months. I've filed the initial PR for the UDF spec:
>>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>>>>>>>>>> consensus we've built and addresses the write amplification concern raised
>>>>>>>>>>>> in our last discussion.
>>>>>>>>>>>>
>>>>>>>>>>>> Please take a look and share your thoughts. Happy to discuss it
>>>>>>>>>>>> further during Monday's meeting as well.
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>> .
>>>>>>>>>>>>>
>>>>>>>>>>>>> We also discussed a way to avoid copying every overload into
>>>>>>>>>>>>> the new metadata JSON when creating a new version. One of ideas is to
>>>>>>>>>>>>> introduce a global version array, this is not yet reflected in the doc, but
>>>>>>>>>>>>> I’ll update it shortly. Other key points:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>>>>>>>    execute.
>>>>>>>>>>>>>    - Keeping the version while referring to an UDF
>>>>>>>>>>>>>    probably isn't a good idea. Users are responsible for updating downstream
>>>>>>>>>>>>>    views if they reference older UDF versions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In general,
>>>>>>>>>>>>>> we discussed the UDF metadata structure, captured at this doc(
>>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    1. Each UDF overload has its own return type. e.g.,
>>>>>>>>>>>>>>    `add(int, int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>>>>>>    2. Return type should be explicitly specified, no
>>>>>>>>>>>>>>    implicit or statement-based return type inference should be allowed.
>>>>>>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>>>>>>    properties at the overload level.
>>>>>>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>>>>>>    5. Introducing a dedicated signature definitions section
>>>>>>>>>>>>>>    to centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>>>>>>    6. Whether to have versioned open properties or not.
>>>>>>>>>>>>>>    Versioned properties can lead to unnecessary copying of a bag of properties
>>>>>>>>>>>>>>    into each version, while it provides a clear history of properties for any
>>>>>>>>>>>>>>    future debugging and understanding of the UDF behavior at a
>>>>>>>>>>>>>>    specific point in time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <
>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>>>>>>>>>> Apologies for the delay!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>>>>>>       - We agreed to combine overloads with the same name
>>>>>>>>>>>>>>>       into a single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>>>>>>       - Listing UDFs will return a list of UDF names, not a
>>>>>>>>>>>>>>>       list of individual signatures.
>>>>>>>>>>>>>>>       - Loading a UDF by name will return all of its
>>>>>>>>>>>>>>>       overloads.
>>>>>>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>>>>>>       - A global version number will track changes across
>>>>>>>>>>>>>>>       the entire UDF entity, it increments monolithically.
>>>>>>>>>>>>>>>       - Each overload will also maintain its own version
>>>>>>>>>>>>>>>       (e.g., updated_at_version) to trace changes specific to that overload.
>>>>>>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>>>>>>    this stage.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs with
>>>>>>>>>>>>>>>> multiple signatures under the same name, which can serve both
>>>>>>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Per my investigation[1], most engines support overloading
>>>>>>>>>>>>>>>> by arguments and allow implicit conversions like numeric widening (e.g.,
>>>>>>>>>>>>>>>> INT → BIGINT/FLOAT). The resolution approach causes issues like silent
>>>>>>>>>>>>>>>> behavior change. Here is an example:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected
>>>>>>>>>>>>>>>>    code.
>>>>>>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>>>>>>    - Engine’s best-match resolution now binds the same
>>>>>>>>>>>>>>>>    call to the new overload, changing behavior without modifying the query.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To mitigate this issue, we have to choose between these two
>>>>>>>>>>>>>>>> access control models:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all
>>>>>>>>>>>>>>>>    overloads of a function name.
>>>>>>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to
>>>>>>>>>>>>>>>>    specific signatures.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The general recommendation is to adopt *Model A.* It
>>>>>>>>>>>>>>>> trades some precision for safety and simplicity, while eliminating the
>>>>>>>>>>>>>>>> silent behavior change problem. More details are in this doc[1].
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>> We have discussed how to define function identifiers
>>>>>>>>>>>>>>>>> (should also handle function overloading). Ryan suggested that we should
>>>>>>>>>>>>>>>>> check how Spark does it. We can refer to functions using an identifier and
>>>>>>>>>>>>>>>>> then bind the different signatures to it. So that access policies can be
>>>>>>>>>>>>>>>>> applied per identifier. This is also linked to how we want to version the
>>>>>>>>>>>>>>>>> functions when overloading is supported.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please check/subscribe to the dev events calendar for the
>>>>>>>>>>>>>>>>> next meeting link (Aug 11).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev
>>>>>>>>>>>>>>>>>> Events" calendar for tomorrow 7/28 at 9AM PT. I missed
>>>>>>>>>>>>>>>>>> the last one, but i'll be at this one.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> No one joined the sync today. I came to know that Yufei
>>>>>>>>>>>>>>>>>>> is on holiday, and Ryan and others couldn't make it, similar to the last
>>>>>>>>>>>>>>>>>>> sync. It seems Yufei might have forgotten to transfer meeting ownership as
>>>>>>>>>>>>>>>>>>> well, as new members needed admin approval and couldn't join automatically
>>>>>>>>>>>>>>>>>>> this week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've updated the function signature schema and other
>>>>>>>>>>>>>>>>>>> open points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a top
>>>>>>>>>>>>>>>>>>>> level field to a field inside “version” along with a representation, so
>>>>>>>>>>>>>>>>>>>> that properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> That said, it's still an open question whether we need
>>>>>>>>>>>>>>>>>>>> any non-versioned properties. We can introduce them later if a use case
>>>>>>>>>>>>>>>>>>>> arises.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from
>>>>>>>>>>>>>>>>>>>>> the representation history. Here are reasons:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    1. Each version still enforces a single signature.
>>>>>>>>>>>>>>>>>>>>>    Although the signatures array is global to the UDF, each version references
>>>>>>>>>>>>>>>>>>>>>    just one signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they
>>>>>>>>>>>>>>>>>>>>>    should remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>> We have discussed the action items from the last sync
>>>>>>>>>>>>>>>>>>>>>> (*see Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the
>>>>>>>>>>>>>>>>>>>>>>    engines and in the roadmaps of many engines. Iceberg will support it. We
>>>>>>>>>>>>>>>>>>>>>>    will maintain the `FunctionIdentifier` (extends `TableIdentifer` but also
>>>>>>>>>>>>>>>>>>>>>>    have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a
>>>>>>>>>>>>>>>>>>>>>>    bag, we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies
>>>>>>>>>>>>>>>>>>>>>>    are supported by some engines. Iceberg will support it. Store the body as
>>>>>>>>>>>>>>>>>>>>>>    it is while creating function by the engine.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    - Standardizing the property names
>>>>>>>>>>>>>>>>>>>>>>    (deterministic, secure).
>>>>>>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level
>>>>>>>>>>>>>>>>>>>>>>    replace is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>>>>>>    - Signature should be associated with
>>>>>>>>>>>>>>>>>>>>>>    representation?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please review
>>>>>>>>>>>>>>>>>>>>>>    the proposal
>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the
>>>>>>>>>>>>>>>>>>>>>>> whole metadata is encrypted, we don't have to worry about just hiding the
>>>>>>>>>>>>>>>>>>>>>>> UDF body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown
>>>>>>>>>>>>>>>>>>>>>>>> are required.We will need a named key(e.g., secure) somewhere, no matter if
>>>>>>>>>>>>>>>>>>>>>>>> it is a top level property or a key as a part of the UDF properties. So
>>>>>>>>>>>>>>>>>>>>>>>> that both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the
>>>>>>>>>>>>>>>>>>>>>>>>> spec would require? Would it require hiding the UDF definition from users
>>>>>>>>>>>>>>>>>>>>>>>>> and require specific pushdown cases be disabled? The use cases seem valid,
>>>>>>>>>>>>>>>>>>>>>>>>> but I'm trying to understand the requirements this places on engines and
>>>>>>>>>>>>>>>>>>>>>>>>> why it needs to be part of the spec, rather than part of the properties of
>>>>>>>>>>>>>>>>>>>>>>>>> the UDF.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes
>>>>>>>>>>>>>>>>>>>>>>>>>>    concealing the UDF body and details like the list of imports, some of them
>>>>>>>>>>>>>>>>>>>>>>>>>>    aren’t applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in
>>>>>>>>>>>>>>>>>>>>>>>>>>    an isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time:
>>>>>>>>>>>>>>>>>>>>>>>>>>    For example, secure UDFs may disable certain optimizations—such as
>>>>>>>>>>>>>>>>>>>>>>>>>>    predicate pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point
>>>>>>>>>>>>>>>>>>>>>>>>>> that the secure flag is primarily an instruction
>>>>>>>>>>>>>>>>>>>>>>>>>> to the engine to behave differently. While it's largely an engine-side
>>>>>>>>>>>>>>>>>>>>>>>>>> behavior, we still need to include this flag in the UDF definition to
>>>>>>>>>>>>>>>>>>>>>>>>>> indicate whether a UDF is secure, especially considering the perf
>>>>>>>>>>>>>>>>>>>>>>>>>> penalty introduced by scenario #3. We should clearly recommend that users
>>>>>>>>>>>>>>>>>>>>>>>>>> avoid marking UDFs as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for
>>>>>>>>>>>>>>>>>>>>>>>>>>> supporting a "secure" UDF? What use case are you addressing and what
>>>>>>>>>>>>>>>>>>>>>>>>>>> specifically changes about how the UDF is handled? If the idea is to hide
>>>>>>>>>>>>>>>>>>>>>>>>>>> the UDF definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted
>>>>>>>>>>>>>>>>>>>>>>>>>>> engine". When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether those statements run within a single transaction should be treated
>>>>>>>>>>>>>>>>>>>>>>>>>>>> as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> “secure UDF” option supported by some engines[1], so the body and any
>>>>>>>>>>>>>>>>>>>>>>>>>>>> sensitive details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    separator, like $$ or '' or none. Action item was to check whether engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    support multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Need to check whether these engines support function overloading for SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDFs. Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    the deterministic flag and concluded that we keep the independent fields
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    (like deterministic) in spec only if the majority of engines supports it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Else it will be passed in a property bag (engine specific). And it is the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    engine's responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> document here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support;
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    the majority agreed *not to* (see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    recording for details).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    will be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    each catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    existing table/view spec — e.g., location
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    will remain *required* for cross-catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sync for the UDF project. Everyone’s welcome to drop in and share ideas!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun.zou to discuss the UDF proposal. We covered several key points, though
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some are still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines may not support vararg syntax in CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION, Iceberg UDFs could represent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently doesn’t support generic types (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> object), we can only map engine-specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as a language for SQL UDFs seems promising, especially given its potential
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> two primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to more interested participants) to continue refining and finalizing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the previous comments. Additionally, I've included the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported by various vendors, including Dremio, Snowflake, Databricks, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deeper discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there was significant interest in storing UDFs in Iceberg and adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations to be SQL. It merely does not specify (in this revision)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (adding a new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement of the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concerns as this proposal focuses on SQL-based engines, while Python-based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> systems often work with data frames. Adding imperative languages like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Python would make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fully understand yet how the proposed design addresses shareability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal and added comments. I think it would be helpful to also have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> design doc that covers the choices from the draft spec. For instance, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> choice to enumerate all possible function input struts rather than allowing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate function signatures is limiting. It would be nice to see a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion of the trade-offs and a rationale for the choice. I think it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would also be very helpful to have a few representative use cases for this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> included in the doc. That way the proposal can demonstrate that it solves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those use cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is inconsistent with conventions in other specs. For example, using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> string IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for spec versioning than the Iceberg view and table specs. It requires
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> readers to fail if there are any unknown fields, which prevents the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from adding things that are fully backward-compatible. Other Iceberg specs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only require a version change to introduce forward-incompatible changes and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think that this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow multiple function signatures per verison, but it is unclear how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encode them because a version is associated with a single function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for creating functions across engines, so this doesn’t show that the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function shows a SELECT statement and it isn’t clear how this is distinct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> no more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is an interesting direction, but there might be some details that need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be fine tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> who might be interested. Resharing since I do not think it was directly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> linked in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> since we didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review so far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> meant scalar/aggregate/table user defined functions. Here are some examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to is a variation of (1) where the API is data flow/data pipeline API
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of SQL (e.g., Spark Scala). Yes, that is also possible in the very
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explore some long term opportunities in this case. Consider you register a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark temp view as some sort of data frame read, then it could still be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolved to a Spark plan that is representable by an intermediate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representation. But I agree this gets very complicated very soon, and just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> having the case (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that a tabular SQL UDF can be used to build a parameterized view.  So,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there's definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initially supports SQL representations of UDFs (similar to views as shared
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the reference links above), the complexity involved will be similar to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> publishing the draft spec (inspired by the view spec) this week to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> facilitate further discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the views? I feel we can say exactly the same thing for Iceberg views,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but yet we have Iceberg multi-dialect views implemented. Maybe it sounds
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like we are trying to draw a line between SQL vs other programming language
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding UDF support is a good idea, even just a multi-dialect one like view,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and that can allow engines to for example parse a view SQL, and when a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 1:32 AM Robert Stupp <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> define a common base upon which engines can build, so the argument that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs aren't practical, because engines are different, is probably only a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temporary concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a negative touch to the idea of having UDFs in Iceberg, especially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether it's a good idea to add UDFs tracked by Iceberg catalogs. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that Iceberg primarily deals with things that are centralized, like tables
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion to gauge the community interest in storing the Versioned SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec addition for storing the versioned UDFs in Iceberg (inspired by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operate similarly to views in that they are associated with tables, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they can accept arguments and produce return values, or even function as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines. Potentially engines can understand the UDFs written by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Thanks everyone for joining today’s UDF sync. We covered several topics:

1. We generally agreed to prefer a simpler form for complex function type
signatures, instead of the Iceberg type JSON format (
https://iceberg.apache.org/spec/#appendix-c-json-serialization).
2. We discussed the motivations and concerns around having a global
parameter name list. Given the trade-offs, we decided to take a deeper look
offline.
3. We talked about default values in terms of overload disambiguation and
cross-engine support, and agreed that this needs further investigation.

Going forward, we’ll use the following Google Doc to record key design
decisions and rationale, so we have a shared reference as discussions
evolve:
https://docs.google.com/document/d/1GKtqFt9gnlJxwo5kfExupDf1jASlgcnDeZv8vEaJQUs/edit?usp=sharing

The meeting recording is available here: https://youtu.be/tF7Ur88snR0

Thanks Kevin for the proposal. Given the holiday season, we'd like to
propose cancelling the next Iceberg UDF Sync scheduled for December 29,
2025. Please let me know if there are any objections.

Yufei


On Fri, Dec 5, 2025 at 8:51 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> Thanks for joining Monday’s UDF sync!
> We covered two updates:
>
>    - Parameter names moved to the top level, this keeps named-argument
>    invocation consistent across overloads.
>    - Default value support added, callers can now omit parameters that
>    have defaults.
>
> The recording is available here for more details:
> https://youtu.be/6CBMwwbISOE
>
> I’ve also added more examples demonstrating default values. Please take a
> look at the PR: https://github.com/apache/iceberg/pull/14117
>
>
> Yufei
>
>
> On Wed, Nov 26, 2025 at 5:57 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Made a few changes to the spec. The PR(
>> https://github.com/apache/iceberg/pull/14117) is ready for another look.
>> 1. Moved the parameter name list to the top level, so that named-argument
>> invocation is consistent across overloads.
>> 2. Added support for default value, which allows caller to ignore
>> parameters with default values.
>>
>> Yufei
>>
>>
>> On Mon, Nov 17, 2025 at 6:42 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Thanks for joining today’s community UDF sync! Here’s a quick recap of
>>> what we discussed and agreed on:
>>>
>>>    1. Huaxin walked us through the Spark POC(
>>>    https://github.com/apache/iceberg/pull/14505). Thanks Huaxin!
>>>    2. When default values are enabled, ambiguous overloads must not be
>>>    allowed. We agreed that definition-id alone is insufficient for conflict
>>>    detection; additional signature-level check across overloads is needed.
>>>    3. We will explore the idea that parameter names are globally
>>>    consistent across multiple overloads. For example, if the overload foo(int
>>>    x, int y) exists, foo(int a) is not allowed, but foo(int x) is OK. We could
>>>    put a list of names and their docs at the top level. This benefits UX and
>>>    may potentially benefit the strategy to pick up the overload to use given
>>>    the inputs. We may bring back the parameter names at the representation
>>>    level, so that it's more flexible.
>>>    4. Should an engine ignore the default values or fail fast if it
>>>    doesn’t support it? We concluded that this is up to the engines, and
>>>    engines could enforce all parameters to be provided when default values
>>>    aren't supported.
>>>    5. How do we give the default value for complex types(e.g. struct)?
>>>    We should follow the way table spec handles the default value(
>>>    https://iceberg.apache.org/spec/#edge-interpolation-algorithm).
>>>
>>> You can watch the recording here for more details:
>>> https://youtu.be/Wxz1nSP48k0
>>>
>>> Please take another look at PR
>>> https://github.com/apache/iceberg/pull/14117 when you get a chance.
>>>
>>> Yufei
>>>
>>>
>>> On Fri, Nov 14, 2025 at 4:29 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> The PR (https://github.com/apache/iceberg/pull/14117) has been
>>>> updated, please take another look.
>>>> For next Monday’s UDF sync, Huaxin will demo a Spark POC showing how
>>>> the current (WIP) UDF spec works. We can also walk through how default
>>>> values should be introduced into the spec.
>>>>
>>>> *TL;DR:* I haven’t added default-value support in the PR yet. It’s
>>>> turning out to be trickier than expected because default values make
>>>> parameters optional. That introduces ambiguity for overloads like:
>>>>
>>>>    -
>>>>
>>>>    foo(int)
>>>>    -
>>>>
>>>>    foo(int, int default 0)
>>>>    -
>>>>
>>>>    foo(int, float default 1.0)
>>>>
>>>> All of these could be invoked as foo(int). The first question is:
>>>> should we allow such overloads to coexist? Probably not, both Postgres and
>>>> Snowflake support default values, but neither allows this form of ambiguous
>>>> overloading.
>>>> Assuming we disallow it, the current UDF definition-id (type tuple)
>>>> isn’t sufficient to detect conflicts. We also need to check the signatures
>>>> of existing overloads to determine whether a new one is valid. For example,
>>>> if (int, int) exists, then neither (int) nor (int, float) should be
>>>> allowed.
>>>>
>>>> Reference:
>>>> Postgres behavior on default parameters:
>>>> https://www.postgresql.org/docs/current/static/typeconv-func.html
>>>> Functions with default arguments match calls that omit zero or more
>>>> defaultable parameters. If multiple functions match, the earliest in the
>>>> search path is chosen; if they are equally specific, Postgres throws an
>>>> “ambiguous function call” error.
>>>> Thanks,
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap of
>>>>> what we discussed and agreed on:
>>>>> 1. We’ll keep parameters only at the definition layer to support
>>>>> named-argument invocation without extra work from the engine side.
>>>>> 2. We agreed to add support for default parameter values.
>>>>> 3. Moving forward with a flag for nullable vs. non-nullable return
>>>>> values, treating it as a hint rather than a strict rule.
>>>>> 4. The udf metadata will preserve the original case of parameter
>>>>> names, and engines can decide whether to normalize or convert them.
>>>>>
>>>>> You can watch the recording here for more details:
>>>>> https://www.youtube.com/watch?v=0i6nKfDNdGw
>>>>> Sorry that today's sync wasn't in the Iceberg Dev Event calendar(It
>>>>> was removed somehow). I have added it back to Iceberg Dev Event as a
>>>>> biweekly sync. Let me know if you have any questions.
>>>>>
>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>> for item 1 and 2 soon. The overall content will remain largely the same.
>>>>> Please take a look when you get a chance!
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Ryan and me are in this meeting, please join
>>>>>>
>>>>>> Iceberg UDF sync
>>>>>> Monday, November 3 · 9:00 – 10:00am
>>>>>> Time zone: America/Los_Angeles
>>>>>> Google Meet joining info
>>>>>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Thanks a lot for the review! Made a few changes to the spec:
>>>>>>> 1. Renamed "overload" to "definition"
>>>>>>> 2. Changed definition id from long to string with the format of
>>>>>>> parameter type tuple.
>>>>>>> 3. Explicitly claimed that named argument invocation is supported
>>>>>>> 4. Added a nullable flag for return values
>>>>>>> 5. Added a few rules for inputs and returns
>>>>>>>
>>>>>>> Please find more details in the PR(
>>>>>>> https://github.com/apache/iceberg/pull/14117). Here is a doc
>>>>>>> comparing engine behaviors in terms of named argument invocation, nullable
>>>>>>> return value, default values for inputs, null handling:
>>>>>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>>>>>> This can help for decision making.
>>>>>>>
>>>>>>> We could discuss the following items in Monday's meeting:
>>>>>>> 1. Whether we need to push down parameters to representation given
>>>>>>> we want to support named argument invocation, it requires extra mapping on
>>>>>>> the engine side.
>>>>>>> 2. Whether we support default values.  Spark plans to support it,
>>>>>>> Snowflake supports it already.
>>>>>>>
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> Thanks for joining today’s community UDF sync. Here’s a summary of
>>>>>>>> what we discussed and agreed upon:
>>>>>>>>
>>>>>>>>    1. We agreed to use Iceberg types for each field in the UDF
>>>>>>>>    spec. This approach ensures consistency with the table spec and allows for
>>>>>>>>    straightforward conversion to JSON types.
>>>>>>>>    2. Using numeric overload IDs can have side effects, for
>>>>>>>>    instance, deleting an overload and later creating a new one with the same
>>>>>>>>    signature will use a different numeric id, which points to the same
>>>>>>>>    overload. The consensus is to represent the overload ID as a string derived
>>>>>>>>    from parameter types and their order.
>>>>>>>>    3. Parameter names must remain consistent across all overload
>>>>>>>>    versions. We agreed to add explicit rules:
>>>>>>>>       1. Parameter names must match across all overload versions.
>>>>>>>>       2. Renaming a parameter requires removing all previous
>>>>>>>>       versions that use the old name.
>>>>>>>>    4. We discussed whether parameters or return types can be
>>>>>>>>    nullable and how engines should handle null values. We’ll research SQL
>>>>>>>>    standards and common engine behaviors before finalizing. The current
>>>>>>>>    leaning is:
>>>>>>>>       1. Allow all parameters to be nullable.
>>>>>>>>       2. Allow the return type to include an explicit nullability
>>>>>>>>       annotation.
>>>>>>>>
>>>>>>>> You can watch the recording here for more details:
>>>>>>>> https://youtu.be/IUd-SN0CQRs
>>>>>>>> Will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>> soon.
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>>>>>>>
>>>>>>>>>    1. Replace “definitions version” with “definitions logs”,
>>>>>>>>>    which record overload changes and timestamps for informational purposes
>>>>>>>>>    only.
>>>>>>>>>    2. Use a globally increasing numeric ID for overloads instead
>>>>>>>>>    of UUIDs.
>>>>>>>>>    3. Add explicit types to all fields in the spec to clarify the
>>>>>>>>>    metadata structure.
>>>>>>>>>    4. Add null-handling optimization hints to help engines skip
>>>>>>>>>    evaluation when inputs are null. (Thanks, Talat!)
>>>>>>>>>    5. Function parameters remain ordered, but we don’t need to
>>>>>>>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>>>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>>>>>    nested structs.
>>>>>>>>>
>>>>>>>>> You can watch the recording here for more details:
>>>>>>>>> https://youtu.be/gJnueQODWvs
>>>>>>>>> I have updated the PR (
>>>>>>>>> https://github.com/apache/iceberg/pull/14117) for the above
>>>>>>>>> feedback. Please take a look and let me know if you have any questions.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>>>>>>>
>>>>>>>>>>    1. Instead of relying on dynamic inference, the return
>>>>>>>>>>    table’s schema for user-defined table functions (UDTFs) should be
>>>>>>>>>>    explicitly defined.
>>>>>>>>>>    2. Whether a function is a UDTF should be captured as a
>>>>>>>>>>    dedicated attribute, rather than being inferred indirectly from the return
>>>>>>>>>>    type.
>>>>>>>>>>    3. The interpretation of a UDF body (whether it is treated as
>>>>>>>>>>    a partial SQL expression or as a full SELECT statement) should be
>>>>>>>>>>    determined by engines. Example: `SELECT x +1` vs. `x + 1`. Different
>>>>>>>>>>    engines have different takes.
>>>>>>>>>>    4. User-defined aggregation functions (UDAFs) are out of
>>>>>>>>>>    scope for now.
>>>>>>>>>>    5. Each overload should include its own current-version
>>>>>>>>>>    field. This avoids relying solely on the global `definition-versions` when
>>>>>>>>>>    querying the current version of one overload.
>>>>>>>>>>
>>>>>>>>>> You can watch the recording here:
>>>>>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>>>> shortly.
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi folks,
>>>>>>>>>>>
>>>>>>>>>>> Really appreciated feedback from you all over the past few
>>>>>>>>>>> months. I've filed the initial PR for the UDF spec:
>>>>>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>>>>>>>>> consensus we've built and addresses the write amplification concern raised
>>>>>>>>>>> in our last discussion.
>>>>>>>>>>>
>>>>>>>>>>> Please take a look and share your thoughts. Happy to discuss it
>>>>>>>>>>> further during Monday's meeting as well.
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>>>>>
>>>>>>>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>> .
>>>>>>>>>>>>
>>>>>>>>>>>> We also discussed a way to avoid copying every overload into
>>>>>>>>>>>> the new metadata JSON when creating a new version. One of ideas is to
>>>>>>>>>>>> introduce a global version array, this is not yet reflected in the doc, but
>>>>>>>>>>>> I’ll update it shortly. Other key points:
>>>>>>>>>>>>
>>>>>>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>>>>>>    execute.
>>>>>>>>>>>>    - Keeping the version while referring to an UDF
>>>>>>>>>>>>    probably isn't a good idea. Users are responsible for updating downstream
>>>>>>>>>>>>    views if they reference older UDF versions.
>>>>>>>>>>>>
>>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In general,
>>>>>>>>>>>>> we discussed the UDF metadata structure, captured at this doc(
>>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1. Each UDF overload has its own return type. e.g.,
>>>>>>>>>>>>>    `add(int, int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>>>>>    2. Return type should be explicitly specified, no implicit
>>>>>>>>>>>>>    or statement-based return type inference should be allowed.
>>>>>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>>>>>    properties at the overload level.
>>>>>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>>>>>    5. Introducing a dedicated signature definitions section
>>>>>>>>>>>>>    to centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>>>>>    6. Whether to have versioned open properties or not.
>>>>>>>>>>>>>    Versioned properties can lead to unnecessary copying of a bag of properties
>>>>>>>>>>>>>    into each version, while it provides a clear history of properties for any
>>>>>>>>>>>>>    future debugging and understanding of the UDF behavior at a
>>>>>>>>>>>>>    specific point in time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>>>>>>>>> Apologies for the delay!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>>>>>       - We agreed to combine overloads with the same name
>>>>>>>>>>>>>>       into a single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>>>>>       - Listing UDFs will return a list of UDF names, not a
>>>>>>>>>>>>>>       list of individual signatures.
>>>>>>>>>>>>>>       - Loading a UDF by name will return all of its
>>>>>>>>>>>>>>       overloads.
>>>>>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>>>>>       - A global version number will track changes across
>>>>>>>>>>>>>>       the entire UDF entity, it increments monolithically.
>>>>>>>>>>>>>>       - Each overload will also maintain its own version
>>>>>>>>>>>>>>       (e.g., updated_at_version) to trace changes specific to that overload.
>>>>>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>>>>>    this stage.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs with
>>>>>>>>>>>>>>> multiple signatures under the same name, which can serve both
>>>>>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Per my investigation[1], most engines support overloading by
>>>>>>>>>>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>>>>>>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>>>>>>>>>>> change. Here is an example:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected
>>>>>>>>>>>>>>>    code.
>>>>>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>>>>>    - Engine’s best-match resolution now binds the same call
>>>>>>>>>>>>>>>    to the new overload, changing behavior without modifying the query.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To mitigate this issue, we have to choose between these two
>>>>>>>>>>>>>>> access control models:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all
>>>>>>>>>>>>>>>    overloads of a function name.
>>>>>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to
>>>>>>>>>>>>>>>    specific signatures.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The general recommendation is to adopt *Model A.* It trades
>>>>>>>>>>>>>>> some precision for safety and simplicity, while eliminating the silent
>>>>>>>>>>>>>>> behavior change problem. More details are in this doc[1].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>> We have discussed how to define function identifiers
>>>>>>>>>>>>>>>> (should also handle function overloading). Ryan suggested that we should
>>>>>>>>>>>>>>>> check how Spark does it. We can refer to functions using an identifier and
>>>>>>>>>>>>>>>> then bind the different signatures to it. So that access policies can be
>>>>>>>>>>>>>>>> applied per identifier. This is also linked to how we want to version the
>>>>>>>>>>>>>>>> functions when overloading is supported.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please check/subscribe to the dev events calendar for the
>>>>>>>>>>>>>>>> next meeting link (Aug 11).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev
>>>>>>>>>>>>>>>>> Events" calendar for tomorrow 7/28 at 9AM PT. I missed
>>>>>>>>>>>>>>>>> the last one, but i'll be at this one.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> No one joined the sync today. I came to know that Yufei
>>>>>>>>>>>>>>>>>> is on holiday, and Ryan and others couldn't make it, similar to the last
>>>>>>>>>>>>>>>>>> sync. It seems Yufei might have forgotten to transfer meeting ownership as
>>>>>>>>>>>>>>>>>> well, as new members needed admin approval and couldn't join automatically
>>>>>>>>>>>>>>>>>> this week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've updated the function signature schema and other open
>>>>>>>>>>>>>>>>>> points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <
>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a top
>>>>>>>>>>>>>>>>>>> level field to a field inside “version” along with a representation, so
>>>>>>>>>>>>>>>>>>> that properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> That said, it's still an open question whether we need
>>>>>>>>>>>>>>>>>>> any non-versioned properties. We can introduce them later if a use case
>>>>>>>>>>>>>>>>>>> arises.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    1. Each version still enforces a single signature.
>>>>>>>>>>>>>>>>>>>>    Although the signatures array is global to the UDF, each version references
>>>>>>>>>>>>>>>>>>>>    just one signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they
>>>>>>>>>>>>>>>>>>>>    should remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the
>>>>>>>>>>>>>>>>>>>>>    engines and in the roadmaps of many engines. Iceberg will support it. We
>>>>>>>>>>>>>>>>>>>>>    will maintain the `FunctionIdentifier` (extends `TableIdentifer` but also
>>>>>>>>>>>>>>>>>>>>>    have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a
>>>>>>>>>>>>>>>>>>>>>    bag, we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies
>>>>>>>>>>>>>>>>>>>>>    are supported by some engines. Iceberg will support it. Store the body as
>>>>>>>>>>>>>>>>>>>>>    it is while creating function by the engine.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    - Standardizing the property names (deterministic,
>>>>>>>>>>>>>>>>>>>>>    secure).
>>>>>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level
>>>>>>>>>>>>>>>>>>>>>    replace is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>>>>>    - Signature should be associated with
>>>>>>>>>>>>>>>>>>>>>    representation?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please review
>>>>>>>>>>>>>>>>>>>>>    the proposal
>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole
>>>>>>>>>>>>>>>>>>>>>> metadata is encrypted, we don't have to worry about just hiding the UDF
>>>>>>>>>>>>>>>>>>>>>> body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown
>>>>>>>>>>>>>>>>>>>>>>> are required.We will need a named key(e.g., secure) somewhere, no matter if
>>>>>>>>>>>>>>>>>>>>>>> it is a top level property or a key as a part of the UDF properties. So
>>>>>>>>>>>>>>>>>>>>>>> that both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the
>>>>>>>>>>>>>>>>>>>>>>>> spec would require? Would it require hiding the UDF definition from users
>>>>>>>>>>>>>>>>>>>>>>>> and require specific pushdown cases be disabled? The use cases seem valid,
>>>>>>>>>>>>>>>>>>>>>>>> but I'm trying to understand the requirements this places on engines and
>>>>>>>>>>>>>>>>>>>>>>>> why it needs to be part of the spec, rather than part of the properties of
>>>>>>>>>>>>>>>>>>>>>>>> the UDF.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes
>>>>>>>>>>>>>>>>>>>>>>>>>    concealing the UDF body and details like the list of imports, some of them
>>>>>>>>>>>>>>>>>>>>>>>>>    aren’t applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in
>>>>>>>>>>>>>>>>>>>>>>>>>    an isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For
>>>>>>>>>>>>>>>>>>>>>>>>>    example, secure UDFs may disable certain optimizations—such as predicate
>>>>>>>>>>>>>>>>>>>>>>>>>    pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point
>>>>>>>>>>>>>>>>>>>>>>>>> that the secure flag is primarily an instruction
>>>>>>>>>>>>>>>>>>>>>>>>> to the engine to behave differently. While it's largely an engine-side
>>>>>>>>>>>>>>>>>>>>>>>>> behavior, we still need to include this flag in the UDF definition to
>>>>>>>>>>>>>>>>>>>>>>>>> indicate whether a UDF is secure, especially considering the perf
>>>>>>>>>>>>>>>>>>>>>>>>> penalty introduced by scenario #3. We should clearly recommend that users
>>>>>>>>>>>>>>>>>>>>>>>>> avoid marking UDFs as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting
>>>>>>>>>>>>>>>>>>>>>>>>>> a "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted
>>>>>>>>>>>>>>>>>>>>>>>>>> engine". When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but
>>>>>>>>>>>>>>>>>>>>>>>>>>> whether those statements run within a single transaction should be treated
>>>>>>>>>>>>>>>>>>>>>>>>>>> as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a
>>>>>>>>>>>>>>>>>>>>>>>>>>> “secure UDF” option supported by some engines[1], so the body and any
>>>>>>>>>>>>>>>>>>>>>>>>>>> sensitive details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    like $$ or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Need to check whether these engines support function overloading for SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDFs. Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    the deterministic flag and concluded that we keep the independent fields
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    (like deterministic) in spec only if the majority of engines supports it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Else it will be passed in a property bag (engine specific). And it is the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    engine's responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>> document here
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    majority agreed *not to* (see recording
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    for details).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    will be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    remain *required* for cross-catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sync for the UDF project. Everyone’s welcome to drop in and share ideas!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines may not support vararg syntax in CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION, Iceberg UDFs could represent such
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently doesn’t support generic types (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> object), we can only map engine-specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as a language for SQL UDFs seems promising, especially given its potential
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to more interested participants) to continue refining and finalizing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the previous comments. Additionally, I've included the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported by various vendors, including Dremio, Snowflake, Databricks, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deeper discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there was significant interest in storing UDFs in Iceberg and adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations to be SQL. It merely does not specify (in this revision)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirement of the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as this proposal focuses on SQL-based engines, while Python-based systems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> often work with data frames. Adding imperative languages like Python would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fully understand yet how the proposed design addresses shareability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal and added comments. I think it would be helpful to also have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> design doc that covers the choices from the draft spec. For instance, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> choice to enumerate all possible function input struts rather than allowing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate function signatures is limiting. It would be nice to see a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion of the trade-offs and a rationale for the choice. I think it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would also be very helpful to have a few representative use cases for this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> included in the doc. That way the proposal can demonstrate that it solves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those use cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is inconsistent with conventions in other specs. For example, using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> string IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for spec versioning than the Iceberg view and table specs. It requires
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> readers to fail if there are any unknown fields, which prevents the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from adding things that are fully backward-compatible. Other Iceberg specs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only require a version change to introduce forward-incompatible changes and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think that this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow multiple function signatures per verison, but it is unclear how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encode them because a version is associated with a single function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for creating functions across engines, so this doesn’t show that the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function shows a SELECT statement and it isn’t clear how this is distinct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is an interesting direction, but there might be some details that need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be fine tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> who might be interested. Resharing since I do not think it was directly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> linked in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> since we didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review so far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> meant scalar/aggregate/table user defined functions. Here are some examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to is a variation of (1) where the API is data flow/data pipeline API
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of SQL (e.g., Spark Scala). Yes, that is also possible in the very
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explore some long term opportunities in this case. Consider you register a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark temp view as some sort of data frame read, then it could still be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolved to a Spark plan that is representable by an intermediate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representation. But I agree this gets very complicated very soon, and just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> having the case (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:40 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that a tabular SQL UDF can be used to build a parameterized view.  So,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there's definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initially supports SQL representations of UDFs (similar to views as shared
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the reference links above), the complexity involved will be similar to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> publishing the draft spec (inspired by the view spec) this week to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> facilitate further discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:33 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the views? I feel we can say exactly the same thing for Iceberg views,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but yet we have Iceberg multi-dialect views implemented. Maybe it sounds
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like we are trying to draw a line between SQL vs other programming language
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDF support is a good idea, even just a multi-dialect one like view, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 1:32 AM Robert Stupp <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a common base upon which engines can build, so the argument that UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aren't practical, because engines are different, is probably only a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temporary concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a negative touch to the idea of having UDFs in Iceberg, especially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether it's a good idea to add UDFs tracked by Iceberg catalogs. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that Iceberg primarily deals with things that are centralized, like tables
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to gauge the community interest in storing the Versioned SQL UDFs in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec addition for storing the versioned UDFs in Iceberg (inspired by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operate similarly to views in that they are associated with tables, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they can accept arguments and produce return values, or even function as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines. Potentially engines can understand the UDFs written by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

Thanks for joining Monday’s UDF sync!
We covered two updates:

   - Parameter names moved to the top level, this keeps named-argument
   invocation consistent across overloads.
   - Default value support added, callers can now omit parameters that have
   defaults.

The recording is available here for more details:
https://youtu.be/6CBMwwbISOE

I’ve also added more examples demonstrating default values. Please take a
look at the PR: https://github.com/apache/iceberg/pull/14117


Yufei


On Wed, Nov 26, 2025 at 5:57 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> Made a few changes to the spec. The PR(
> https://github.com/apache/iceberg/pull/14117) is ready for another look.
> 1. Moved the parameter name list to the top level, so that named-argument
> invocation is consistent across overloads.
> 2. Added support for default value, which allows caller to ignore
> parameters with default values.
>
> Yufei
>
>
> On Mon, Nov 17, 2025 at 6:42 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks for joining today’s community UDF sync! Here’s a quick recap of
>> what we discussed and agreed on:
>>
>>    1. Huaxin walked us through the Spark POC(
>>    https://github.com/apache/iceberg/pull/14505). Thanks Huaxin!
>>    2. When default values are enabled, ambiguous overloads must not be
>>    allowed. We agreed that definition-id alone is insufficient for conflict
>>    detection; additional signature-level check across overloads is needed.
>>    3. We will explore the idea that parameter names are globally
>>    consistent across multiple overloads. For example, if the overload foo(int
>>    x, int y) exists, foo(int a) is not allowed, but foo(int x) is OK. We could
>>    put a list of names and their docs at the top level. This benefits UX and
>>    may potentially benefit the strategy to pick up the overload to use given
>>    the inputs. We may bring back the parameter names at the representation
>>    level, so that it's more flexible.
>>    4. Should an engine ignore the default values or fail fast if it
>>    doesn’t support it? We concluded that this is up to the engines, and
>>    engines could enforce all parameters to be provided when default values
>>    aren't supported.
>>    5. How do we give the default value for complex types(e.g. struct)?
>>    We should follow the way table spec handles the default value(
>>    https://iceberg.apache.org/spec/#edge-interpolation-algorithm).
>>
>> You can watch the recording here for more details:
>> https://youtu.be/Wxz1nSP48k0
>>
>> Please take another look at PR
>> https://github.com/apache/iceberg/pull/14117 when you get a chance.
>>
>> Yufei
>>
>>
>> On Fri, Nov 14, 2025 at 4:29 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> The PR (https://github.com/apache/iceberg/pull/14117) has been updated,
>>> please take another look.
>>> For next Monday’s UDF sync, Huaxin will demo a Spark POC showing how the
>>> current (WIP) UDF spec works. We can also walk through how default values
>>> should be introduced into the spec.
>>>
>>> *TL;DR:* I haven’t added default-value support in the PR yet. It’s
>>> turning out to be trickier than expected because default values make
>>> parameters optional. That introduces ambiguity for overloads like:
>>>
>>>    -
>>>
>>>    foo(int)
>>>    -
>>>
>>>    foo(int, int default 0)
>>>    -
>>>
>>>    foo(int, float default 1.0)
>>>
>>> All of these could be invoked as foo(int). The first question is:
>>> should we allow such overloads to coexist? Probably not, both Postgres and
>>> Snowflake support default values, but neither allows this form of ambiguous
>>> overloading.
>>> Assuming we disallow it, the current UDF definition-id (type tuple)
>>> isn’t sufficient to detect conflicts. We also need to check the signatures
>>> of existing overloads to determine whether a new one is valid. For example,
>>> if (int, int) exists, then neither (int) nor (int, float) should be
>>> allowed.
>>>
>>> Reference:
>>> Postgres behavior on default parameters:
>>> https://www.postgresql.org/docs/current/static/typeconv-func.html
>>> Functions with default arguments match calls that omit zero or more
>>> defaultable parameters. If multiple functions match, the earliest in the
>>> search path is chosen; if they are equally specific, Postgres throws an
>>> “ambiguous function call” error.
>>> Thanks,
>>> Yufei
>>>
>>>
>>> On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Thanks for joining today’s community UDF sync! Here’s a quick recap of
>>>> what we discussed and agreed on:
>>>> 1. We’ll keep parameters only at the definition layer to support
>>>> named-argument invocation without extra work from the engine side.
>>>> 2. We agreed to add support for default parameter values.
>>>> 3. Moving forward with a flag for nullable vs. non-nullable return
>>>> values, treating it as a hint rather than a strict rule.
>>>> 4. The udf metadata will preserve the original case of parameter names,
>>>> and engines can decide whether to normalize or convert them.
>>>>
>>>> You can watch the recording here for more details:
>>>> https://www.youtube.com/watch?v=0i6nKfDNdGw
>>>> Sorry that today's sync wasn't in the Iceberg Dev Event calendar(It was
>>>> removed somehow). I have added it back to Iceberg Dev Event as a biweekly
>>>> sync. Let me know if you have any questions.
>>>>
>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117) for
>>>> item 1 and 2 soon. The overall content will remain largely the same. Please
>>>> take a look when you get a chance!
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Ryan and me are in this meeting, please join
>>>>>
>>>>> Iceberg UDF sync
>>>>> Monday, November 3 · 9:00 – 10:00am
>>>>> Time zone: America/Los_Angeles
>>>>> Google Meet joining info
>>>>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> Thanks a lot for the review! Made a few changes to the spec:
>>>>>> 1. Renamed "overload" to "definition"
>>>>>> 2. Changed definition id from long to string with the format of
>>>>>> parameter type tuple.
>>>>>> 3. Explicitly claimed that named argument invocation is supported
>>>>>> 4. Added a nullable flag for return values
>>>>>> 5. Added a few rules for inputs and returns
>>>>>>
>>>>>> Please find more details in the PR(
>>>>>> https://github.com/apache/iceberg/pull/14117). Here is a doc
>>>>>> comparing engine behaviors in terms of named argument invocation, nullable
>>>>>> return value, default values for inputs, null handling:
>>>>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>>>>> This can help for decision making.
>>>>>>
>>>>>> We could discuss the following items in Monday's meeting:
>>>>>> 1. Whether we need to push down parameters to representation given we
>>>>>> want to support named argument invocation, it requires extra mapping on the
>>>>>> engine side.
>>>>>> 2. Whether we support default values.  Spark plans to support it,
>>>>>> Snowflake supports it already.
>>>>>>
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Thanks for joining today’s community UDF sync. Here’s a summary of
>>>>>>> what we discussed and agreed upon:
>>>>>>>
>>>>>>>    1. We agreed to use Iceberg types for each field in the UDF
>>>>>>>    spec. This approach ensures consistency with the table spec and allows for
>>>>>>>    straightforward conversion to JSON types.
>>>>>>>    2. Using numeric overload IDs can have side effects, for
>>>>>>>    instance, deleting an overload and later creating a new one with the same
>>>>>>>    signature will use a different numeric id, which points to the same
>>>>>>>    overload. The consensus is to represent the overload ID as a string derived
>>>>>>>    from parameter types and their order.
>>>>>>>    3. Parameter names must remain consistent across all overload
>>>>>>>    versions. We agreed to add explicit rules:
>>>>>>>       1. Parameter names must match across all overload versions.
>>>>>>>       2. Renaming a parameter requires removing all previous
>>>>>>>       versions that use the old name.
>>>>>>>    4. We discussed whether parameters or return types can be
>>>>>>>    nullable and how engines should handle null values. We’ll research SQL
>>>>>>>    standards and common engine behaviors before finalizing. The current
>>>>>>>    leaning is:
>>>>>>>       1. Allow all parameters to be nullable.
>>>>>>>       2. Allow the return type to include an explicit nullability
>>>>>>>       annotation.
>>>>>>>
>>>>>>> You can watch the recording here for more details:
>>>>>>> https://youtu.be/IUd-SN0CQRs
>>>>>>> Will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>> soon.
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>>>>>>
>>>>>>>>    1. Replace “definitions version” with “definitions logs”, which
>>>>>>>>    record overload changes and timestamps for informational purposes only.
>>>>>>>>    2. Use a globally increasing numeric ID for overloads instead
>>>>>>>>    of UUIDs.
>>>>>>>>    3. Add explicit types to all fields in the spec to clarify the
>>>>>>>>    metadata structure.
>>>>>>>>    4. Add null-handling optimization hints to help engines skip
>>>>>>>>    evaluation when inputs are null. (Thanks, Talat!)
>>>>>>>>    5. Function parameters remain ordered, but we don’t need to
>>>>>>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>>>>    nested structs.
>>>>>>>>
>>>>>>>> You can watch the recording here for more details:
>>>>>>>> https://youtu.be/gJnueQODWvs
>>>>>>>> I have updated the PR (https://github.com/apache/iceberg/pull/14117)
>>>>>>>> for the above feedback. Please take a look and let me know if you have any
>>>>>>>> questions.
>>>>>>>>
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>>>>>>
>>>>>>>>>    1. Instead of relying on dynamic inference, the return table’s
>>>>>>>>>    schema for user-defined table functions (UDTFs) should be explicitly
>>>>>>>>>    defined.
>>>>>>>>>    2. Whether a function is a UDTF should be captured as a
>>>>>>>>>    dedicated attribute, rather than being inferred indirectly from the return
>>>>>>>>>    type.
>>>>>>>>>    3. The interpretation of a UDF body (whether it is treated as
>>>>>>>>>    a partial SQL expression or as a full SELECT statement) should be
>>>>>>>>>    determined by engines. Example: `SELECT x +1` vs. `x + 1`. Different
>>>>>>>>>    engines have different takes.
>>>>>>>>>    4. User-defined aggregation functions (UDAFs) are out of scope
>>>>>>>>>    for now.
>>>>>>>>>    5. Each overload should include its own current-version field.
>>>>>>>>>    This avoids relying solely on the global `definition-versions` when
>>>>>>>>>    querying the current version of one overload.
>>>>>>>>>
>>>>>>>>> You can watch the recording here:
>>>>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>>> shortly.
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> Really appreciated feedback from you all over the past few
>>>>>>>>>> months. I've filed the initial PR for the UDF spec:
>>>>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>>>>>>>> consensus we've built and addresses the write amplification concern raised
>>>>>>>>>> in our last discussion.
>>>>>>>>>>
>>>>>>>>>> Please take a look and share your thoughts. Happy to discuss it
>>>>>>>>>> further during Monday's meeting as well.
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>>>>
>>>>>>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>> We also discussed a way to avoid copying every overload into the
>>>>>>>>>>> new metadata JSON when creating a new version. One of ideas is to introduce
>>>>>>>>>>> a global version array, this is not yet reflected in the doc, but I’ll
>>>>>>>>>>> update it shortly. Other key points:
>>>>>>>>>>>
>>>>>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>>>>>    execute.
>>>>>>>>>>>    - Keeping the version while referring to an UDF
>>>>>>>>>>>    probably isn't a good idea. Users are responsible for updating downstream
>>>>>>>>>>>    views if they reference older UDF versions.
>>>>>>>>>>>
>>>>>>>>>>> You can watch the recording here:
>>>>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>>>>>>>>>>> discussed the UDF metadata structure, captured at this doc(
>>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>>>>
>>>>>>>>>>>>    1. Each UDF overload has its own return type. e.g.,
>>>>>>>>>>>>    `add(int, int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>>>>    2. Return type should be explicitly specified, no implicit
>>>>>>>>>>>>    or statement-based return type inference should be allowed.
>>>>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>>>>    properties at the overload level.
>>>>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>>>>    5. Introducing a dedicated signature definitions section to
>>>>>>>>>>>>    centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>>>>    6. Whether to have versioned open properties or not.
>>>>>>>>>>>>    Versioned properties can lead to unnecessary copying of a bag of properties
>>>>>>>>>>>>    into each version, while it provides a clear history of properties for any
>>>>>>>>>>>>    future debugging and understanding of the UDF behavior at a
>>>>>>>>>>>>    specific point in time.
>>>>>>>>>>>>
>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>>>>>>>> Apologies for the delay!
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>>>>       - We agreed to combine overloads with the same name
>>>>>>>>>>>>>       into a single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>>>>       - Listing UDFs will return a list of UDF names, not a
>>>>>>>>>>>>>       list of individual signatures.
>>>>>>>>>>>>>       - Loading a UDF by name will return all of its
>>>>>>>>>>>>>       overloads.
>>>>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>>>>       - A global version number will track changes across the
>>>>>>>>>>>>>       entire UDF entity, it increments monolithically.
>>>>>>>>>>>>>       - Each overload will also maintain its own version
>>>>>>>>>>>>>       (e.g., updated_at_version) to trace changes specific to that overload.
>>>>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>>>>    this stage.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs with
>>>>>>>>>>>>>> multiple signatures under the same name, which can serve both
>>>>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Per my investigation[1], most engines support overloading by
>>>>>>>>>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>>>>>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>>>>>>>>>> change. Here is an example:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>>>>    - Engine’s best-match resolution now binds the same call
>>>>>>>>>>>>>>    to the new overload, changing behavior without modifying the query.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To mitigate this issue, we have to choose between these two
>>>>>>>>>>>>>> access control models:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all
>>>>>>>>>>>>>>    overloads of a function name.
>>>>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>>>>>>>>>>    signatures.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The general recommendation is to adopt *Model A.* It trades
>>>>>>>>>>>>>> some precision for safety and simplicity, while eliminating the silent
>>>>>>>>>>>>>> behavior change problem. More details are in this doc[1].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>> We have discussed how to define function identifiers (should
>>>>>>>>>>>>>>> also handle function overloading). Ryan suggested that we should check how
>>>>>>>>>>>>>>> Spark does it. We can refer to functions using an identifier and then bind
>>>>>>>>>>>>>>> the different signatures to it. So that access policies can be applied per
>>>>>>>>>>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>>>>>>>>>>> when overloading is supported.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please check/subscribe to the dev events calendar for the
>>>>>>>>>>>>>>> next meeting link (Aug 11).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev
>>>>>>>>>>>>>>>> Events" calendar for tomorrow 7/28 at 9AM PT. I missed the
>>>>>>>>>>>>>>>> last one, but i'll be at this one.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> No one joined the sync today. I came to know that Yufei is
>>>>>>>>>>>>>>>>> on holiday, and Ryan and others couldn't make it, similar to the last sync.
>>>>>>>>>>>>>>>>> It seems Yufei might have forgotten to transfer meeting ownership as well,
>>>>>>>>>>>>>>>>> as new members needed admin approval and couldn't join automatically this
>>>>>>>>>>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've updated the function signature schema and other open
>>>>>>>>>>>>>>>>> points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <
>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a top
>>>>>>>>>>>>>>>>>> level field to a field inside “version” along with a representation, so
>>>>>>>>>>>>>>>>>> that properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> That said, it's still an open question whether we need
>>>>>>>>>>>>>>>>>> any non-versioned properties. We can introduce them later if a use case
>>>>>>>>>>>>>>>>>> arises.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    1. Each version still enforces a single signature.
>>>>>>>>>>>>>>>>>>>    Although the signatures array is global to the UDF, each version references
>>>>>>>>>>>>>>>>>>>    just one signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they
>>>>>>>>>>>>>>>>>>>    should remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the
>>>>>>>>>>>>>>>>>>>>    engines and in the roadmaps of many engines. Iceberg will support it. We
>>>>>>>>>>>>>>>>>>>>    will maintain the `FunctionIdentifier` (extends `TableIdentifer` but also
>>>>>>>>>>>>>>>>>>>>    have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a
>>>>>>>>>>>>>>>>>>>>    bag, we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies
>>>>>>>>>>>>>>>>>>>>    are supported by some engines. Iceberg will support it. Store the body as
>>>>>>>>>>>>>>>>>>>>    it is while creating function by the engine.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    - Standardizing the property names (deterministic,
>>>>>>>>>>>>>>>>>>>>    secure).
>>>>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level
>>>>>>>>>>>>>>>>>>>>    replace is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>>>>    - Signature should be associated with
>>>>>>>>>>>>>>>>>>>>    representation?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please review the
>>>>>>>>>>>>>>>>>>>>    proposal
>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole
>>>>>>>>>>>>>>>>>>>>> metadata is encrypted, we don't have to worry about just hiding the UDF
>>>>>>>>>>>>>>>>>>>>> body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the
>>>>>>>>>>>>>>>>>>>>>>> spec would require? Would it require hiding the UDF definition from users
>>>>>>>>>>>>>>>>>>>>>>> and require specific pushdown cases be disabled? The use cases seem valid,
>>>>>>>>>>>>>>>>>>>>>>> but I'm trying to understand the requirements this places on engines and
>>>>>>>>>>>>>>>>>>>>>>> why it needs to be part of the spec, rather than part of the properties of
>>>>>>>>>>>>>>>>>>>>>>> the UDF.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes
>>>>>>>>>>>>>>>>>>>>>>>>    concealing the UDF body and details like the list of imports, some of them
>>>>>>>>>>>>>>>>>>>>>>>>    aren’t applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in
>>>>>>>>>>>>>>>>>>>>>>>>    an isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For
>>>>>>>>>>>>>>>>>>>>>>>>    example, secure UDFs may disable certain optimizations—such as predicate
>>>>>>>>>>>>>>>>>>>>>>>>    pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that
>>>>>>>>>>>>>>>>>>>>>>>> the secure flag is primarily an instruction to the
>>>>>>>>>>>>>>>>>>>>>>>> engine to behave differently. While it's largely an engine-side behavior,
>>>>>>>>>>>>>>>>>>>>>>>> we still need to include this flag in the UDF definition to indicate
>>>>>>>>>>>>>>>>>>>>>>>> whether a UDF is secure, especially considering the perf penalty introduced
>>>>>>>>>>>>>>>>>>>>>>>> by scenario #3. We should clearly recommend that users avoid marking UDFs
>>>>>>>>>>>>>>>>>>>>>>>> as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting
>>>>>>>>>>>>>>>>>>>>>>>>> a "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted
>>>>>>>>>>>>>>>>>>>>>>>>> engine". When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but
>>>>>>>>>>>>>>>>>>>>>>>>>> whether those statements run within a single transaction should be treated
>>>>>>>>>>>>>>>>>>>>>>>>>> as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a
>>>>>>>>>>>>>>>>>>>>>>>>>> “secure UDF” option supported by some engines[1], so the body and any
>>>>>>>>>>>>>>>>>>>>>>>>>> sensitive details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator,
>>>>>>>>>>>>>>>>>>>>>>>>>>>    like $$ or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need
>>>>>>>>>>>>>>>>>>>>>>>>>>>    to check whether these engines support function overloading for SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed
>>>>>>>>>>>>>>>>>>>>>>>>>>>    the deterministic flag and concluded that we keep the independent fields
>>>>>>>>>>>>>>>>>>>>>>>>>>>    (like deterministic) in spec only if the majority of engines supports it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    Else it will be passed in a property bag (engine specific). And it is the
>>>>>>>>>>>>>>>>>>>>>>>>>>>    engine's responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>> document here
>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once
>>>>>>>>>>>>>>>>>>>>>>>>>>> it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    majority agreed *not to* (see recording for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    details).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    will be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    remain *required* for cross-catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sync for the UDF project. Everyone’s welcome to drop in and share ideas!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines may not support vararg syntax in CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION, Iceberg UDFs could represent such
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently doesn’t support generic types (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> object), we can only map engine-specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the previous comments. Additionally, I've included the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported by various vendors, including Dremio, Snowflake, Databricks, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deeper discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there was significant interest in storing UDFs in Iceberg and adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations to be SQL. It merely does not specify (in this revision)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as this proposal focuses on SQL-based engines, while Python-based systems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> often work with data frames. Adding imperative languages like Python would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fully understand yet how the proposed design addresses shareability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and added comments. I think it would be helpful to also have a design doc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate function signatures is limiting. It would be nice to see a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion of the trade-offs and a rationale for the choice. I think it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would also be very helpful to have a few representative use cases for this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> included in the doc. That way the proposal can demonstrate that it solves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those use cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is inconsistent with conventions in other specs. For example, using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> string IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for spec versioning than the Iceberg view and table specs. It requires
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> readers to fail if there are any unknown fields, which prevents the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from adding things that are fully backward-compatible. Other Iceberg specs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only require a version change to introduce forward-incompatible changes and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think that this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow multiple function signatures per verison, but it is unclear how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encode them because a version is associated with a single function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for creating functions across engines, so this doesn’t show that the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function shows a SELECT statement and it isn’t clear how this is distinct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an interesting direction, but there might be some details that need to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fine tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> who might be interested. Resharing since I do not think it was directly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> linked in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> meant scalar/aggregate/table user defined functions. Here are some examples
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to is a variation of (1) where the API is data flow/data pipeline API
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of SQL (e.g., Spark Scala). Yes, that is also possible in the very
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explore some long term opportunities in this case. Consider you register a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark temp view as some sort of data frame read, then it could still be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolved to a Spark plan that is representable by an intermediate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representation. But I agree this gets very complicated very soon, and just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> having the case (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:40 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that a tabular SQL UDF can be used to build a parameterized view.  So,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there's definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initially supports SQL representations of UDFs (similar to views as shared
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the reference links above), the complexity involved will be similar to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> publishing the draft spec (inspired by the view spec) this week to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> facilitate further discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:33 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the views? I feel we can say exactly the same thing for Iceberg views,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but yet we have Iceberg multi-dialect views implemented. Maybe it sounds
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like we are trying to draw a line between SQL vs other programming language
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDF support is a good idea, even just a multi-dialect one like view, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:32 AM Robert Stupp <sn...@snazy.de>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a common base upon which engines can build, so the argument that UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aren't practical, because engines are different, is probably only a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temporary concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a negative touch to the idea of having UDFs in Iceberg, especially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether it's a good idea to add UDFs tracked by Iceberg catalogs. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that Iceberg primarily deals with things that are centralized, like tables
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when you post the proposal, but I think this would be a very difficult
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> area to tackle across engines, languages, and memory models without having
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to gauge the community interest in storing the Versioned SQL UDFs in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec addition for storing the versioned UDFs in Iceberg (inspired by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operate similarly to views in that they are associated with tables, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they can accept arguments and produce return values, or even function as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines. Potentially engines can understand the UDFs written by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

Made a few changes to the spec. The PR(
https://github.com/apache/iceberg/pull/14117) is ready for another look.
1. Moved the parameter name list to the top level, so that named-argument
invocation is consistent across overloads.
2. Added support for default value, which allows caller to ignore
parameters with default values.

Yufei


On Mon, Nov 17, 2025 at 6:42 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> Thanks for joining today’s community UDF sync! Here’s a quick recap of
> what we discussed and agreed on:
>
>    1. Huaxin walked us through the Spark POC(
>    https://github.com/apache/iceberg/pull/14505). Thanks Huaxin!
>    2. When default values are enabled, ambiguous overloads must not be
>    allowed. We agreed that definition-id alone is insufficient for conflict
>    detection; additional signature-level check across overloads is needed.
>    3. We will explore the idea that parameter names are globally
>    consistent across multiple overloads. For example, if the overload foo(int
>    x, int y) exists, foo(int a) is not allowed, but foo(int x) is OK. We could
>    put a list of names and their docs at the top level. This benefits UX and
>    may potentially benefit the strategy to pick up the overload to use given
>    the inputs. We may bring back the parameter names at the representation
>    level, so that it's more flexible.
>    4. Should an engine ignore the default values or fail fast if it
>    doesn’t support it? We concluded that this is up to the engines, and
>    engines could enforce all parameters to be provided when default values
>    aren't supported.
>    5. How do we give the default value for complex types(e.g. struct)? We
>    should follow the way table spec handles the default value(
>    https://iceberg.apache.org/spec/#edge-interpolation-algorithm).
>
> You can watch the recording here for more details:
> https://youtu.be/Wxz1nSP48k0
>
> Please take another look at PR
> https://github.com/apache/iceberg/pull/14117 when you get a chance.
>
> Yufei
>
>
> On Fri, Nov 14, 2025 at 4:29 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> The PR (https://github.com/apache/iceberg/pull/14117) has been updated,
>> please take another look.
>> For next Monday’s UDF sync, Huaxin will demo a Spark POC showing how the
>> current (WIP) UDF spec works. We can also walk through how default values
>> should be introduced into the spec.
>>
>> *TL;DR:* I haven’t added default-value support in the PR yet. It’s
>> turning out to be trickier than expected because default values make
>> parameters optional. That introduces ambiguity for overloads like:
>>
>>    -
>>
>>    foo(int)
>>    -
>>
>>    foo(int, int default 0)
>>    -
>>
>>    foo(int, float default 1.0)
>>
>> All of these could be invoked as foo(int). The first question is: should
>> we allow such overloads to coexist? Probably not, both Postgres and
>> Snowflake support default values, but neither allows this form of ambiguous
>> overloading.
>> Assuming we disallow it, the current UDF definition-id (type tuple) isn’t
>> sufficient to detect conflicts. We also need to check the signatures of
>> existing overloads to determine whether a new one is valid. For example, if
>> (int, int) exists, then neither (int) nor (int, float) should be allowed.
>>
>> Reference:
>> Postgres behavior on default parameters:
>> https://www.postgresql.org/docs/current/static/typeconv-func.html
>> Functions with default arguments match calls that omit zero or more
>> defaultable parameters. If multiple functions match, the earliest in the
>> search path is chosen; if they are equally specific, Postgres throws an
>> “ambiguous function call” error.
>> Thanks,
>> Yufei
>>
>>
>> On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Thanks for joining today’s community UDF sync! Here’s a quick recap of
>>> what we discussed and agreed on:
>>> 1. We’ll keep parameters only at the definition layer to support
>>> named-argument invocation without extra work from the engine side.
>>> 2. We agreed to add support for default parameter values.
>>> 3. Moving forward with a flag for nullable vs. non-nullable return
>>> values, treating it as a hint rather than a strict rule.
>>> 4. The udf metadata will preserve the original case of parameter names,
>>> and engines can decide whether to normalize or convert them.
>>>
>>> You can watch the recording here for more details:
>>> https://www.youtube.com/watch?v=0i6nKfDNdGw
>>> Sorry that today's sync wasn't in the Iceberg Dev Event calendar(It was
>>> removed somehow). I have added it back to Iceberg Dev Event as a biweekly
>>> sync. Let me know if you have any questions.
>>>
>>> I will update the PR(https://github.com/apache/iceberg/pull/14117) for
>>> item 1 and 2 soon. The overall content will remain largely the same. Please
>>> take a look when you get a chance!
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Ryan and me are in this meeting, please join
>>>>
>>>> Iceberg UDF sync
>>>> Monday, November 3 · 9:00 – 10:00am
>>>> Time zone: America/Los_Angeles
>>>> Google Meet joining info
>>>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Thanks a lot for the review! Made a few changes to the spec:
>>>>> 1. Renamed "overload" to "definition"
>>>>> 2. Changed definition id from long to string with the format of
>>>>> parameter type tuple.
>>>>> 3. Explicitly claimed that named argument invocation is supported
>>>>> 4. Added a nullable flag for return values
>>>>> 5. Added a few rules for inputs and returns
>>>>>
>>>>> Please find more details in the PR(
>>>>> https://github.com/apache/iceberg/pull/14117). Here is a doc
>>>>> comparing engine behaviors in terms of named argument invocation, nullable
>>>>> return value, default values for inputs, null handling:
>>>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>>>> This can help for decision making.
>>>>>
>>>>> We could discuss the following items in Monday's meeting:
>>>>> 1. Whether we need to push down parameters to representation given we
>>>>> want to support named argument invocation, it requires extra mapping on the
>>>>> engine side.
>>>>> 2. Whether we support default values.  Spark plans to support it,
>>>>> Snowflake supports it already.
>>>>>
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> Thanks for joining today’s community UDF sync. Here’s a summary of
>>>>>> what we discussed and agreed upon:
>>>>>>
>>>>>>    1. We agreed to use Iceberg types for each field in the UDF spec.
>>>>>>    This approach ensures consistency with the table spec and allows for
>>>>>>    straightforward conversion to JSON types.
>>>>>>    2. Using numeric overload IDs can have side effects, for
>>>>>>    instance, deleting an overload and later creating a new one with the same
>>>>>>    signature will use a different numeric id, which points to the same
>>>>>>    overload. The consensus is to represent the overload ID as a string derived
>>>>>>    from parameter types and their order.
>>>>>>    3. Parameter names must remain consistent across all overload
>>>>>>    versions. We agreed to add explicit rules:
>>>>>>       1. Parameter names must match across all overload versions.
>>>>>>       2. Renaming a parameter requires removing all previous
>>>>>>       versions that use the old name.
>>>>>>    4. We discussed whether parameters or return types can be
>>>>>>    nullable and how engines should handle null values. We’ll research SQL
>>>>>>    standards and common engine behaviors before finalizing. The current
>>>>>>    leaning is:
>>>>>>       1. Allow all parameters to be nullable.
>>>>>>       2. Allow the return type to include an explicit nullability
>>>>>>       annotation.
>>>>>>
>>>>>> You can watch the recording here for more details:
>>>>>> https://youtu.be/IUd-SN0CQRs
>>>>>> Will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>> soon.
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>>>>>
>>>>>>>    1. Replace “definitions version” with “definitions logs”, which
>>>>>>>    record overload changes and timestamps for informational purposes only.
>>>>>>>    2. Use a globally increasing numeric ID for overloads instead of
>>>>>>>    UUIDs.
>>>>>>>    3. Add explicit types to all fields in the spec to clarify the
>>>>>>>    metadata structure.
>>>>>>>    4. Add null-handling optimization hints to help engines skip
>>>>>>>    evaluation when inputs are null. (Thanks, Talat!)
>>>>>>>    5. Function parameters remain ordered, but we don’t need to
>>>>>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>>>    nested structs.
>>>>>>>
>>>>>>> You can watch the recording here for more details:
>>>>>>> https://youtu.be/gJnueQODWvs
>>>>>>> I have updated the PR (https://github.com/apache/iceberg/pull/14117)
>>>>>>> for the above feedback. Please take a look and let me know if you have any
>>>>>>> questions.
>>>>>>>
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>>>>>
>>>>>>>>    1. Instead of relying on dynamic inference, the return table’s
>>>>>>>>    schema for user-defined table functions (UDTFs) should be explicitly
>>>>>>>>    defined.
>>>>>>>>    2. Whether a function is a UDTF should be captured as a
>>>>>>>>    dedicated attribute, rather than being inferred indirectly from the return
>>>>>>>>    type.
>>>>>>>>    3. The interpretation of a UDF body (whether it is treated as a
>>>>>>>>    partial SQL expression or as a full SELECT statement) should be determined
>>>>>>>>    by engines. Example: `SELECT x +1` vs. `x + 1`. Different engines have
>>>>>>>>    different takes.
>>>>>>>>    4. User-defined aggregation functions (UDAFs) are out of scope
>>>>>>>>    for now.
>>>>>>>>    5. Each overload should include its own current-version field.
>>>>>>>>    This avoids relying solely on the global `definition-versions` when
>>>>>>>>    querying the current version of one overload.
>>>>>>>>
>>>>>>>> You can watch the recording here:
>>>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>>> shortly.
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> Really appreciated feedback from you all over the past few months.
>>>>>>>>> I've filed the initial PR for the UDF spec:
>>>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>>>>>>> consensus we've built and addresses the write amplification concern raised
>>>>>>>>> in our last discussion.
>>>>>>>>>
>>>>>>>>> Please take a look and share your thoughts. Happy to discuss it
>>>>>>>>> further during Monday's meeting as well.
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>>>
>>>>>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>>> We also discussed a way to avoid copying every overload into the
>>>>>>>>>> new metadata JSON when creating a new version. One of ideas is to introduce
>>>>>>>>>> a global version array, this is not yet reflected in the doc, but I’ll
>>>>>>>>>> update it shortly. Other key points:
>>>>>>>>>>
>>>>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>>>>    execute.
>>>>>>>>>>    - Keeping the version while referring to an UDF
>>>>>>>>>>    probably isn't a good idea. Users are responsible for updating downstream
>>>>>>>>>>    views if they reference older UDF versions.
>>>>>>>>>>
>>>>>>>>>> You can watch the recording here:
>>>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>>>>>>>>>> discussed the UDF metadata structure, captured at this doc(
>>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>>>
>>>>>>>>>>>    1. Each UDF overload has its own return type. e.g.,
>>>>>>>>>>>    `add(int, int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>>>    2. Return type should be explicitly specified, no implicit
>>>>>>>>>>>    or statement-based return type inference should be allowed.
>>>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>>>    properties at the overload level.
>>>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>>>    5. Introducing a dedicated signature definitions section to
>>>>>>>>>>>    centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>>>    6. Whether to have versioned open properties or not.
>>>>>>>>>>>    Versioned properties can lead to unnecessary copying of a bag of properties
>>>>>>>>>>>    into each version, while it provides a clear history of properties for any
>>>>>>>>>>>    future debugging and understanding of the UDF behavior at a
>>>>>>>>>>>    specific point in time.
>>>>>>>>>>>
>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>>>>>>> Apologies for the delay!
>>>>>>>>>>>>
>>>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>>>       - We agreed to combine overloads with the same name into
>>>>>>>>>>>>       a single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>>>       - Listing UDFs will return a list of UDF names, not a
>>>>>>>>>>>>       list of individual signatures.
>>>>>>>>>>>>       - Loading a UDF by name will return all of its overloads.
>>>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>>>       - A global version number will track changes across the
>>>>>>>>>>>>       entire UDF entity, it increments monolithically.
>>>>>>>>>>>>       - Each overload will also maintain its own version
>>>>>>>>>>>>       (e.g., updated_at_version) to trace changes specific to that overload.
>>>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>>>    this stage.
>>>>>>>>>>>>
>>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs with
>>>>>>>>>>>>> multiple signatures under the same name, which can serve both
>>>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Per my investigation[1], most engines support overloading by
>>>>>>>>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>>>>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>>>>>>>>> change. Here is an example:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>>>    - Engine’s best-match resolution now binds the same call
>>>>>>>>>>>>>    to the new overload, changing behavior without modifying the query.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To mitigate this issue, we have to choose between these two
>>>>>>>>>>>>> access control models:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads
>>>>>>>>>>>>>    of a function name.
>>>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>>>>>>>>>    signatures.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The general recommendation is to adopt *Model A.* It trades
>>>>>>>>>>>>> some precision for safety and simplicity, while eliminating the silent
>>>>>>>>>>>>> behavior change problem. More details are in this doc[1].
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1.
>>>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>> We have discussed how to define function identifiers (should
>>>>>>>>>>>>>> also handle function overloading). Ryan suggested that we should check how
>>>>>>>>>>>>>> Spark does it. We can refer to functions using an identifier and then bind
>>>>>>>>>>>>>> the different signatures to it. So that access policies can be applied per
>>>>>>>>>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>>>>>>>>>> when overloading is supported.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please check/subscribe to the dev events calendar for the
>>>>>>>>>>>>>> next meeting link (Aug 11).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev
>>>>>>>>>>>>>>> Events" calendar for tomorrow 7/28 at 9AM PT. I missed the
>>>>>>>>>>>>>>> last one, but i'll be at this one.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> No one joined the sync today. I came to know that Yufei is
>>>>>>>>>>>>>>>> on holiday, and Ryan and others couldn't make it, similar to the last sync.
>>>>>>>>>>>>>>>> It seems Yufei might have forgotten to transfer meeting ownership as well,
>>>>>>>>>>>>>>>> as new members needed admin approval and couldn't join automatically this
>>>>>>>>>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've updated the function signature schema and other open
>>>>>>>>>>>>>>>> points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a top
>>>>>>>>>>>>>>>>> level field to a field inside “version” along with a representation, so
>>>>>>>>>>>>>>>>> that properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>>>>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    1. Each version still enforces a single signature.
>>>>>>>>>>>>>>>>>>    Although the signatures array is global to the UDF, each version references
>>>>>>>>>>>>>>>>>>    just one signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they
>>>>>>>>>>>>>>>>>>    should remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the
>>>>>>>>>>>>>>>>>>>    engines and in the roadmaps of many engines. Iceberg will support it. We
>>>>>>>>>>>>>>>>>>>    will maintain the `FunctionIdentifier` (extends `TableIdentifer` but also
>>>>>>>>>>>>>>>>>>>    have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag,
>>>>>>>>>>>>>>>>>>>    we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    - Standardizing the property names (deterministic,
>>>>>>>>>>>>>>>>>>>    secure).
>>>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level replace
>>>>>>>>>>>>>>>>>>>    is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please review the
>>>>>>>>>>>>>>>>>>>    proposal
>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole
>>>>>>>>>>>>>>>>>>>> metadata is encrypted, we don't have to worry about just hiding the UDF
>>>>>>>>>>>>>>>>>>>> body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the
>>>>>>>>>>>>>>>>>>>>>> spec would require? Would it require hiding the UDF definition from users
>>>>>>>>>>>>>>>>>>>>>> and require specific pushdown cases be disabled? The use cases seem valid,
>>>>>>>>>>>>>>>>>>>>>> but I'm trying to understand the requirements this places on engines and
>>>>>>>>>>>>>>>>>>>>>> why it needs to be part of the spec, rather than part of the properties of
>>>>>>>>>>>>>>>>>>>>>> the UDF.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing
>>>>>>>>>>>>>>>>>>>>>>>    the UDF body and details like the list of imports, some of them aren’t
>>>>>>>>>>>>>>>>>>>>>>>    applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an
>>>>>>>>>>>>>>>>>>>>>>>    isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For
>>>>>>>>>>>>>>>>>>>>>>>    example, secure UDFs may disable certain optimizations—such as predicate
>>>>>>>>>>>>>>>>>>>>>>>    pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that
>>>>>>>>>>>>>>>>>>>>>>> the secure flag is primarily an instruction to the
>>>>>>>>>>>>>>>>>>>>>>> engine to behave differently. While it's largely an engine-side behavior,
>>>>>>>>>>>>>>>>>>>>>>> we still need to include this flag in the UDF definition to indicate
>>>>>>>>>>>>>>>>>>>>>>> whether a UDF is secure, especially considering the perf penalty introduced
>>>>>>>>>>>>>>>>>>>>>>> by scenario #3. We should clearly recommend that users avoid marking UDFs
>>>>>>>>>>>>>>>>>>>>>>> as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a
>>>>>>>>>>>>>>>>>>>>>>>> "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted
>>>>>>>>>>>>>>>>>>>>>>>> engine". When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but
>>>>>>>>>>>>>>>>>>>>>>>>> whether those statements run within a single transaction should be treated
>>>>>>>>>>>>>>>>>>>>>>>>> as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure
>>>>>>>>>>>>>>>>>>>>>>>>> UDF” option supported by some engines[1], so the body and any sensitive
>>>>>>>>>>>>>>>>>>>>>>>>> details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator,
>>>>>>>>>>>>>>>>>>>>>>>>>>    like $$ or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need
>>>>>>>>>>>>>>>>>>>>>>>>>>    to check whether these engines support function overloading for SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>    Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed
>>>>>>>>>>>>>>>>>>>>>>>>>>    the deterministic flag and concluded that we keep the independent fields
>>>>>>>>>>>>>>>>>>>>>>>>>>    (like deterministic) in spec only if the majority of engines supports it.
>>>>>>>>>>>>>>>>>>>>>>>>>>    Else it will be passed in a property bag (engine specific). And it is the
>>>>>>>>>>>>>>>>>>>>>>>>>>    engine's responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal document
>>>>>>>>>>>>>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it
>>>>>>>>>>>>>>>>>>>>>>>>>> is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the
>>>>>>>>>>>>>>>>>>>>>>>>>>>    majority agreed *not to* (see recording for
>>>>>>>>>>>>>>>>>>>>>>>>>>>    details).
>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will
>>>>>>>>>>>>>>>>>>>>>>>>>>>    be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each
>>>>>>>>>>>>>>>>>>>>>>>>>>>    catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will remain
>>>>>>>>>>>>>>>>>>>>>>>>>>>    *required* for cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote
>>>>>>>>>>>>>>>>>>>>>>>>>>>    once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community
>>>>>>>>>>>>>>>>>>>>>>>>>>>> sync for the UDF project. Everyone’s welcome to drop in and share ideas!
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines may not support vararg syntax in CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION, Iceberg UDFs could represent such
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently doesn’t support generic types (e.g.,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> object), we can only map engine-specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the previous comments. Additionally, I've included the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported by various vendors, including Dremio, Snowflake, Databricks, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deeper discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there was significant interest in storing UDFs in Iceberg and adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoints for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations to be SQL. It merely does not specify (in this revision)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as this proposal focuses on SQL-based engines, while Python-based systems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> often work with data frames. Adding imperative languages like Python would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fully understand yet how the proposed design addresses shareability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and added comments. I think it would be helpful to also have a design doc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate function signatures is limiting. It would be nice to see a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussion of the trade-offs and a rationale for the choice. I think it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would also be very helpful to have a few representative use cases for this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> included in the doc. That way the proposal can demonstrate that it solves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those use cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is inconsistent with conventions in other specs. For example, using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> string IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for spec versioning than the Iceberg view and table specs. It requires
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> readers to fail if there are any unknown fields, which prevents the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from adding things that are fully backward-compatible. Other Iceberg specs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only require a version change to introduce forward-incompatible changes and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think that this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow multiple function signatures per verison, but it is unclear how to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> encode them because a version is associated with a single function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for creating functions across engines, so this doesn’t show that the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function shows a SELECT statement and it isn’t clear how this is distinct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an interesting direction, but there might be some details that need to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fine tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> who might be interested. Resharing since I do not think it was directly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> linked in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a variation of (1) where the API is data flow/data pipeline API instead
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of SQL (e.g., Spark Scala). Yes, that is also possible in the very long run
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explore some long term opportunities in this case. Consider you register a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark temp view as some sort of data frame read, then it could still be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolved to a Spark plan that is representable by an intermediate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representation. But I agree this gets very complicated very soon, and just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> having the case (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:40 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that a tabular SQL UDF can be used to build a parameterized view.  So,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there's definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initially supports SQL representations of UDFs (similar to views as shared
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the reference links above), the complexity involved will be similar to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> publishing the draft spec (inspired by the view spec) this week to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> facilitate further discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:33 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the views? I feel we can say exactly the same thing for Iceberg views,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but yet we have Iceberg multi-dialect views implemented. Maybe it sounds
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like we are trying to draw a line between SQL vs other programming language
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDF support is a good idea, even just a multi-dialect one like view, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:32 AM Robert Stupp <sn...@snazy.de>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> common base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a negative touch to the idea of having UDFs in Iceberg, especially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether it's a good idea to add UDFs tracked by Iceberg catalogs. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that Iceberg primarily deals with things that are centralized, like tables
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to gauge the community interest in storing the Versioned SQL UDFs in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec addition for storing the versioned UDFs in Iceberg (inspired by view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines. Potentially engines can understand the UDFs written by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

Thanks for joining today’s community UDF sync! Here’s a quick recap of what
we discussed and agreed on:

   1. Huaxin walked us through the Spark POC(
   https://github.com/apache/iceberg/pull/14505). Thanks Huaxin!
   2. When default values are enabled, ambiguous overloads must not be
   allowed. We agreed that definition-id alone is insufficient for conflict
   detection; additional signature-level check across overloads is needed.
   3. We will explore the idea that parameter names are globally consistent
   across multiple overloads. For example, if the overload foo(int x, int y)
   exists, foo(int a) is not allowed, but foo(int x) is OK. We could put a
   list of names and their docs at the top level. This benefits UX and may
   potentially benefit the strategy to pick up the overload to use given the
   inputs. We may bring back the parameter names at the representation level,
   so that it's more flexible.
   4. Should an engine ignore the default values or fail fast if it doesn’t
   support it? We concluded that this is up to the engines, and engines could
   enforce all parameters to be provided when default values aren't supported.
   5. How do we give the default value for complex types(e.g. struct)? We
   should follow the way table spec handles the default value(
   https://iceberg.apache.org/spec/#edge-interpolation-algorithm).

You can watch the recording here for more details:
https://youtu.be/Wxz1nSP48k0

Please take another look at PR
https://github.com/apache/iceberg/pull/14117 when
you get a chance.

Yufei


On Fri, Nov 14, 2025 at 4:29 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> The PR (https://github.com/apache/iceberg/pull/14117) has been updated,
> please take another look.
> For next Monday’s UDF sync, Huaxin will demo a Spark POC showing how the
> current (WIP) UDF spec works. We can also walk through how default values
> should be introduced into the spec.
>
> *TL;DR:* I haven’t added default-value support in the PR yet. It’s
> turning out to be trickier than expected because default values make
> parameters optional. That introduces ambiguity for overloads like:
>
>    -
>
>    foo(int)
>    -
>
>    foo(int, int default 0)
>    -
>
>    foo(int, float default 1.0)
>
> All of these could be invoked as foo(int). The first question is: should
> we allow such overloads to coexist? Probably not, both Postgres and
> Snowflake support default values, but neither allows this form of ambiguous
> overloading.
> Assuming we disallow it, the current UDF definition-id (type tuple) isn’t
> sufficient to detect conflicts. We also need to check the signatures of
> existing overloads to determine whether a new one is valid. For example, if
> (int, int) exists, then neither (int) nor (int, float) should be allowed.
>
> Reference:
> Postgres behavior on default parameters:
> https://www.postgresql.org/docs/current/static/typeconv-func.html
> Functions with default arguments match calls that omit zero or more
> defaultable parameters. If multiple functions match, the earliest in the
> search path is chosen; if they are equally specific, Postgres throws an
> “ambiguous function call” error.
> Thanks,
> Yufei
>
>
> On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks for joining today’s community UDF sync! Here’s a quick recap of
>> what we discussed and agreed on:
>> 1. We’ll keep parameters only at the definition layer to support
>> named-argument invocation without extra work from the engine side.
>> 2. We agreed to add support for default parameter values.
>> 3. Moving forward with a flag for nullable vs. non-nullable return
>> values, treating it as a hint rather than a strict rule.
>> 4. The udf metadata will preserve the original case of parameter names,
>> and engines can decide whether to normalize or convert them.
>>
>> You can watch the recording here for more details:
>> https://www.youtube.com/watch?v=0i6nKfDNdGw
>> Sorry that today's sync wasn't in the Iceberg Dev Event calendar(It was
>> removed somehow). I have added it back to Iceberg Dev Event as a biweekly
>> sync. Let me know if you have any questions.
>>
>> I will update the PR(https://github.com/apache/iceberg/pull/14117) for
>> item 1 and 2 soon. The overall content will remain largely the same. Please
>> take a look when you get a chance!
>>
>> Yufei
>>
>>
>> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Ryan and me are in this meeting, please join
>>>
>>> Iceberg UDF sync
>>> Monday, November 3 · 9:00 – 10:00am
>>> Time zone: America/Los_Angeles
>>> Google Meet joining info
>>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>>
>>> Yufei
>>>
>>>
>>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Thanks a lot for the review! Made a few changes to the spec:
>>>> 1. Renamed "overload" to "definition"
>>>> 2. Changed definition id from long to string with the format of
>>>> parameter type tuple.
>>>> 3. Explicitly claimed that named argument invocation is supported
>>>> 4. Added a nullable flag for return values
>>>> 5. Added a few rules for inputs and returns
>>>>
>>>> Please find more details in the PR(
>>>> https://github.com/apache/iceberg/pull/14117). Here is a doc comparing
>>>> engine behaviors in terms of named argument invocation, nullable return
>>>> value, default values for inputs, null handling:
>>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>>> This can help for decision making.
>>>>
>>>> We could discuss the following items in Monday's meeting:
>>>> 1. Whether we need to push down parameters to representation given we
>>>> want to support named argument invocation, it requires extra mapping on the
>>>> engine side.
>>>> 2. Whether we support default values.  Spark plans to support it,
>>>> Snowflake supports it already.
>>>>
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Thanks for joining today’s community UDF sync. Here’s a summary of
>>>>> what we discussed and agreed upon:
>>>>>
>>>>>    1. We agreed to use Iceberg types for each field in the UDF spec.
>>>>>    This approach ensures consistency with the table spec and allows for
>>>>>    straightforward conversion to JSON types.
>>>>>    2. Using numeric overload IDs can have side effects, for instance,
>>>>>    deleting an overload and later creating a new one with the same signature
>>>>>    will use a different numeric id, which points to the same overload. The
>>>>>    consensus is to represent the overload ID as a string derived from
>>>>>    parameter types and their order.
>>>>>    3. Parameter names must remain consistent across all overload
>>>>>    versions. We agreed to add explicit rules:
>>>>>       1. Parameter names must match across all overload versions.
>>>>>       2. Renaming a parameter requires removing all previous versions
>>>>>       that use the old name.
>>>>>    4. We discussed whether parameters or return types can be nullable
>>>>>    and how engines should handle null values. We’ll research SQL standards and
>>>>>    common engine behaviors before finalizing. The current leaning is:
>>>>>       1. Allow all parameters to be nullable.
>>>>>       2. Allow the return type to include an explicit nullability
>>>>>       annotation.
>>>>>
>>>>> You can watch the recording here for more details:
>>>>> https://youtu.be/IUd-SN0CQRs
>>>>> Will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>> soon.
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>>>>
>>>>>>    1. Replace “definitions version” with “definitions logs”, which
>>>>>>    record overload changes and timestamps for informational purposes only.
>>>>>>    2. Use a globally increasing numeric ID for overloads instead of
>>>>>>    UUIDs.
>>>>>>    3. Add explicit types to all fields in the spec to clarify the
>>>>>>    metadata structure.
>>>>>>    4. Add null-handling optimization hints to help engines skip
>>>>>>    evaluation when inputs are null. (Thanks, Talat!)
>>>>>>    5. Function parameters remain ordered, but we don’t need to
>>>>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>>    nested structs.
>>>>>>
>>>>>> You can watch the recording here for more details:
>>>>>> https://youtu.be/gJnueQODWvs
>>>>>> I have updated the PR (https://github.com/apache/iceberg/pull/14117)
>>>>>> for the above feedback. Please take a look and let me know if you have any
>>>>>> questions.
>>>>>>
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>>>>
>>>>>>>    1. Instead of relying on dynamic inference, the return table’s
>>>>>>>    schema for user-defined table functions (UDTFs) should be explicitly
>>>>>>>    defined.
>>>>>>>    2. Whether a function is a UDTF should be captured as a
>>>>>>>    dedicated attribute, rather than being inferred indirectly from the return
>>>>>>>    type.
>>>>>>>    3. The interpretation of a UDF body (whether it is treated as a
>>>>>>>    partial SQL expression or as a full SELECT statement) should be determined
>>>>>>>    by engines. Example: `SELECT x +1` vs. `x + 1`. Different engines have
>>>>>>>    different takes.
>>>>>>>    4. User-defined aggregation functions (UDAFs) are out of scope
>>>>>>>    for now.
>>>>>>>    5. Each overload should include its own current-version field.
>>>>>>>    This avoids relying solely on the global `definition-versions` when
>>>>>>>    querying the current version of one overload.
>>>>>>>
>>>>>>> You can watch the recording here:
>>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>>> shortly.
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> Really appreciated feedback from you all over the past few months.
>>>>>>>> I've filed the initial PR for the UDF spec:
>>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>>>>>> consensus we've built and addresses the write amplification concern raised
>>>>>>>> in our last discussion.
>>>>>>>>
>>>>>>>> Please take a look and share your thoughts. Happy to discuss it
>>>>>>>> further during Monday's meeting as well.
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>>
>>>>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>> .
>>>>>>>>>
>>>>>>>>> We also discussed a way to avoid copying every overload into the
>>>>>>>>> new metadata JSON when creating a new version. One of ideas is to introduce
>>>>>>>>> a global version array, this is not yet reflected in the doc, but I’ll
>>>>>>>>> update it shortly. Other key points:
>>>>>>>>>
>>>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>>>    execute.
>>>>>>>>>    - Keeping the version while referring to an UDF probably isn't
>>>>>>>>>    a good idea. Users are responsible for updating downstream views if they
>>>>>>>>>    reference older UDF versions.
>>>>>>>>>
>>>>>>>>> You can watch the recording here:
>>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>>>>>>>>> discussed the UDF metadata structure, captured at this doc(
>>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>>
>>>>>>>>>>    1. Each UDF overload has its own return type. e.g., `add(int,
>>>>>>>>>>    int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>>    2. Return type should be explicitly specified, no implicit or
>>>>>>>>>>    statement-based return type inference should be allowed.
>>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>>    properties at the overload level.
>>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>>    5. Introducing a dedicated signature definitions section to
>>>>>>>>>>    centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>>    6. Whether to have versioned open properties or not.
>>>>>>>>>>    Versioned properties can lead to unnecessary copying of a bag of properties
>>>>>>>>>>    into each version, while it provides a clear history of properties for any
>>>>>>>>>>    future debugging and understanding of the UDF behavior at a
>>>>>>>>>>    specific point in time.
>>>>>>>>>>
>>>>>>>>>> Watch the recording here,
>>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>>>>>> Apologies for the delay!
>>>>>>>>>>>
>>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>>       - We agreed to combine overloads with the same name into
>>>>>>>>>>>       a single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>>       - Listing UDFs will return a list of UDF names, not a
>>>>>>>>>>>       list of individual signatures.
>>>>>>>>>>>       - Loading a UDF by name will return all of its overloads.
>>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>>       - A global version number will track changes across the
>>>>>>>>>>>       entire UDF entity, it increments monolithically.
>>>>>>>>>>>       - Each overload will also maintain its own version (e.g.,
>>>>>>>>>>>       updated_at_version) to trace changes specific to that overload.
>>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>>    this stage.
>>>>>>>>>>>
>>>>>>>>>>> Watch the recording here,
>>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs with
>>>>>>>>>>>> multiple signatures under the same name, which can serve both
>>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>>
>>>>>>>>>>>> Per my investigation[1], most engines support overloading by
>>>>>>>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>>>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>>>>>>>> change. Here is an example:
>>>>>>>>>>>>
>>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>>    - Engine’s best-match resolution now binds the same call to
>>>>>>>>>>>>    the new overload, changing behavior without modifying the query.
>>>>>>>>>>>>
>>>>>>>>>>>> To mitigate this issue, we have to choose between these two
>>>>>>>>>>>> access control models:
>>>>>>>>>>>>
>>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads
>>>>>>>>>>>>    of a function name.
>>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>>>>>>>>    signatures.
>>>>>>>>>>>>
>>>>>>>>>>>> The general recommendation is to adopt *Model A.* It trades
>>>>>>>>>>>> some precision for safety and simplicity, while eliminating the silent
>>>>>>>>>>>> behavior change problem. More details are in this doc[1].
>>>>>>>>>>>>
>>>>>>>>>>>> 1.
>>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>>
>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>> We have discussed how to define function identifiers (should
>>>>>>>>>>>>> also handle function overloading). Ryan suggested that we should check how
>>>>>>>>>>>>> Spark does it. We can refer to functions using an identifier and then bind
>>>>>>>>>>>>> the different signatures to it. So that access policies can be applied per
>>>>>>>>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>>>>>>>>> when overloading is supported.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please check/subscribe to the dev events calendar for the next
>>>>>>>>>>>>> meeting link (Aug 11).
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev
>>>>>>>>>>>>>> Events" calendar for tomorrow 7/28 at 9AM PT. I missed the
>>>>>>>>>>>>>> last one, but i'll be at this one.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> No one joined the sync today. I came to know that Yufei is
>>>>>>>>>>>>>>> on holiday, and Ryan and others couldn't make it, similar to the last sync.
>>>>>>>>>>>>>>> It seems Yufei might have forgotten to transfer meeting ownership as well,
>>>>>>>>>>>>>>> as new members needed admin approval and couldn't join automatically this
>>>>>>>>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've updated the function signature schema and other open
>>>>>>>>>>>>>>> points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a top level
>>>>>>>>>>>>>>>> field to a field inside “version” along with a representation, so that
>>>>>>>>>>>>>>>> properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>>>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    1. Each version still enforces a single signature.
>>>>>>>>>>>>>>>>>    Although the signatures array is global to the UDF, each version references
>>>>>>>>>>>>>>>>>    just one signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they
>>>>>>>>>>>>>>>>>    should remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the
>>>>>>>>>>>>>>>>>>    engines and in the roadmaps of many engines. Iceberg will support it. We
>>>>>>>>>>>>>>>>>>    will maintain the `FunctionIdentifier` (extends `TableIdentifer` but also
>>>>>>>>>>>>>>>>>>    have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag,
>>>>>>>>>>>>>>>>>>    we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - Standardizing the property names (deterministic,
>>>>>>>>>>>>>>>>>>    secure).
>>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level replace
>>>>>>>>>>>>>>>>>>    is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please review the
>>>>>>>>>>>>>>>>>>    proposal
>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole
>>>>>>>>>>>>>>>>>>> metadata is encrypted, we don't have to worry about just hiding the UDF
>>>>>>>>>>>>>>>>>>> body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the
>>>>>>>>>>>>>>>>>>>>> spec would require? Would it require hiding the UDF definition from users
>>>>>>>>>>>>>>>>>>>>> and require specific pushdown cases be disabled? The use cases seem valid,
>>>>>>>>>>>>>>>>>>>>> but I'm trying to understand the requirements this places on engines and
>>>>>>>>>>>>>>>>>>>>> why it needs to be part of the spec, rather than part of the properties of
>>>>>>>>>>>>>>>>>>>>> the UDF.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing
>>>>>>>>>>>>>>>>>>>>>>    the UDF body and details like the list of imports, some of them aren’t
>>>>>>>>>>>>>>>>>>>>>>    applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an
>>>>>>>>>>>>>>>>>>>>>>    isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For
>>>>>>>>>>>>>>>>>>>>>>    example, secure UDFs may disable certain optimizations—such as predicate
>>>>>>>>>>>>>>>>>>>>>>    pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that
>>>>>>>>>>>>>>>>>>>>>> the secure flag is primarily an instruction to the
>>>>>>>>>>>>>>>>>>>>>> engine to behave differently. While it's largely an engine-side behavior,
>>>>>>>>>>>>>>>>>>>>>> we still need to include this flag in the UDF definition to indicate
>>>>>>>>>>>>>>>>>>>>>> whether a UDF is secure, especially considering the perf penalty introduced
>>>>>>>>>>>>>>>>>>>>>> by scenario #3. We should clearly recommend that users avoid marking UDFs
>>>>>>>>>>>>>>>>>>>>>> as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a
>>>>>>>>>>>>>>>>>>>>>>> "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted
>>>>>>>>>>>>>>>>>>>>>>> engine". When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but
>>>>>>>>>>>>>>>>>>>>>>>> whether those statements run within a single transaction should be treated
>>>>>>>>>>>>>>>>>>>>>>>> as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure
>>>>>>>>>>>>>>>>>>>>>>>> UDF” option supported by some engines[1], so the body and any sensitive
>>>>>>>>>>>>>>>>>>>>>>>> details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator,
>>>>>>>>>>>>>>>>>>>>>>>>>    like $$ or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need
>>>>>>>>>>>>>>>>>>>>>>>>>    to check whether these engines support function overloading for SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>    Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal document
>>>>>>>>>>>>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it
>>>>>>>>>>>>>>>>>>>>>>>>> is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the
>>>>>>>>>>>>>>>>>>>>>>>>>>    majority agreed *not to* (see recording for
>>>>>>>>>>>>>>>>>>>>>>>>>>    details).
>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will
>>>>>>>>>>>>>>>>>>>>>>>>>>    be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each
>>>>>>>>>>>>>>>>>>>>>>>>>>    catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will remain
>>>>>>>>>>>>>>>>>>>>>>>>>>    *required* for cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote
>>>>>>>>>>>>>>>>>>>>>>>>>>    once it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community
>>>>>>>>>>>>>>>>>>>>>>>>>>> sync for the UDF project. Everyone’s welcome to drop in and share ideas!
>>>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>> may not support vararg syntax in CREATE
>>>>>>>>>>>>>>>>>>>>>>>>>>>> FUNCTION, Iceberg UDFs could represent such
>>>>>>>>>>>>>>>>>>>>>>>>>>>> arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently
>>>>>>>>>>>>>>>>>>>>>>>>>>>> doesn’t support generic types (e.g., object),
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can only map engine-specific types to Iceberg types. As a result,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> generic data types will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two
>>>>>>>>>>>>>>>>>>>>>>>>>>>> primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> more interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the previous comments. Additionally, I've included the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported by various vendors, including Dremio, Snowflake, Databricks, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deeper discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was significant interest in storing UDFs in Iceberg and adding endpoints
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations to be SQL. It merely does not specify (in this revision)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this proposal focuses on SQL-based engines, while Python-based systems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> often work with data frames. Adding imperative languages like Python would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fully understand yet how the proposed design addresses shareability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue <bl...@databricks.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and added comments. I think it would be helpful to also have a design doc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures is limiting. It would be nice to see a discussion of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the trade-offs and a rationale for the choice. I think it would also be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very helpful to have a few representative use cases for this included in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc. That way the proposal can demonstrate that it solves those use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is inconsistent with conventions in other specs. For example, using string
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec versioning than the Iceberg view and table specs. It requires readers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to fail if there are any unknown fields, which prevents the spec from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding things that are fully backward-compatible. Other Iceberg specs only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> require a version change to introduce forward-incompatible changes and I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think that this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple function signatures per verison, but it is unclear how to encode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating functions across engines, so this doesn’t show that the metadata
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function shows a SELECT statement and it isn’t clear how this is distinct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> please have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an interesting direction, but there might be some details that need to be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fine tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might be interested. Resharing since I do not think it was directly linked
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a variation of (1) where the API is data flow/data pipeline API instead
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of SQL (e.g., Spark Scala). Yes, that is also possible in the very long run
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2:57 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explore some long term opportunities in this case. Consider you register a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark temp view as some sort of data frame read, then it could still be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolved to a Spark plan that is representable by an intermediate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representation. But I agree this gets very complicated very soon, and just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> having the case (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:40 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that a tabular SQL UDF can be used to build a parameterized view.  So,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there's definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports SQL representations of UDFs (similar to views as shared by the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reference links above), the complexity involved will be similar to managing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:33 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to have a common set of functions across engines, I don't see how that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the views? I feel we can say exactly the same thing for Iceberg views,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but yet we have Iceberg multi-dialect views implemented. Maybe it sounds
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like we are trying to draw a line between SQL vs other programming language
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDF support is a good idea, even just a multi-dialect one like view, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:32 AM Robert Stupp <sn...@snazy.de>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> common base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a negative touch to the idea of having UDFs in Iceberg, especially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether it's a good idea to add UDFs tracked by Iceberg catalogs. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that Iceberg primarily deals with things that are centralized, like tables
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gauge the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec addition for storing the versioned UDFs in Iceberg (inspired by view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines. Potentially engines can understand the UDFs written by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

The PR (https://github.com/apache/iceberg/pull/14117) has been updated,
please take another look.
For next Monday’s UDF sync, Huaxin will demo a Spark POC showing how the
current (WIP) UDF spec works. We can also walk through how default values
should be introduced into the spec.

*TL;DR:* I haven’t added default-value support in the PR yet. It’s turning
out to be trickier than expected because default values make parameters
optional. That introduces ambiguity for overloads like:

   -

   foo(int)
   -

   foo(int, int default 0)
   -

   foo(int, float default 1.0)

All of these could be invoked as foo(int). The first question is: should we
allow such overloads to coexist? Probably not, both Postgres and Snowflake
support default values, but neither allows this form of ambiguous
overloading.
Assuming we disallow it, the current UDF definition-id (type tuple) isn’t
sufficient to detect conflicts. We also need to check the signatures of
existing overloads to determine whether a new one is valid. For example, if
(int, int) exists, then neither (int) nor (int, float) should be allowed.

Reference:
Postgres behavior on default parameters:
https://www.postgresql.org/docs/current/static/typeconv-func.html
Functions with default arguments match calls that omit zero or more
defaultable parameters. If multiple functions match, the earliest in the
search path is chosen; if they are equally specific, Postgres throws an
“ambiguous function call” error.
Thanks,
Yufei


On Mon, Nov 3, 2025 at 3:34 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> Thanks for joining today’s community UDF sync! Here’s a quick recap of
> what we discussed and agreed on:
> 1. We’ll keep parameters only at the definition layer to support
> named-argument invocation without extra work from the engine side.
> 2. We agreed to add support for default parameter values.
> 3. Moving forward with a flag for nullable vs. non-nullable return values,
> treating it as a hint rather than a strict rule.
> 4. The udf metadata will preserve the original case of parameter names,
> and engines can decide whether to normalize or convert them.
>
> You can watch the recording here for more details:
> https://www.youtube.com/watch?v=0i6nKfDNdGw
> Sorry that today's sync wasn't in the Iceberg Dev Event calendar(It was
> removed somehow). I have added it back to Iceberg Dev Event as a biweekly
> sync. Let me know if you have any questions.
>
> I will update the PR(https://github.com/apache/iceberg/pull/14117) for
> item 1 and 2 soon. The overall content will remain largely the same. Please
> take a look when you get a chance!
>
> Yufei
>
>
> On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> Ryan and me are in this meeting, please join
>>
>> Iceberg UDF sync
>> Monday, November 3 · 9:00 – 10:00am
>> Time zone: America/Los_Angeles
>> Google Meet joining info
>> Video call link: https://meet.google.com/siq-jvtk-tdo
>>
>> Yufei
>>
>>
>> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Thanks a lot for the review! Made a few changes to the spec:
>>> 1. Renamed "overload" to "definition"
>>> 2. Changed definition id from long to string with the format of
>>> parameter type tuple.
>>> 3. Explicitly claimed that named argument invocation is supported
>>> 4. Added a nullable flag for return values
>>> 5. Added a few rules for inputs and returns
>>>
>>> Please find more details in the PR(
>>> https://github.com/apache/iceberg/pull/14117). Here is a doc comparing
>>> engine behaviors in terms of named argument invocation, nullable return
>>> value, default values for inputs, null handling:
>>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>>> This can help for decision making.
>>>
>>> We could discuss the following items in Monday's meeting:
>>> 1. Whether we need to push down parameters to representation given we
>>> want to support named argument invocation, it requires extra mapping on the
>>> engine side.
>>> 2. Whether we support default values.  Spark plans to support it,
>>> Snowflake supports it already.
>>>
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Thanks for joining today’s community UDF sync. Here’s a summary of what
>>>> we discussed and agreed upon:
>>>>
>>>>    1. We agreed to use Iceberg types for each field in the UDF spec.
>>>>    This approach ensures consistency with the table spec and allows for
>>>>    straightforward conversion to JSON types.
>>>>    2. Using numeric overload IDs can have side effects, for instance,
>>>>    deleting an overload and later creating a new one with the same signature
>>>>    will use a different numeric id, which points to the same overload. The
>>>>    consensus is to represent the overload ID as a string derived from
>>>>    parameter types and their order.
>>>>    3. Parameter names must remain consistent across all overload
>>>>    versions. We agreed to add explicit rules:
>>>>       1. Parameter names must match across all overload versions.
>>>>       2. Renaming a parameter requires removing all previous versions
>>>>       that use the old name.
>>>>    4. We discussed whether parameters or return types can be nullable
>>>>    and how engines should handle null values. We’ll research SQL standards and
>>>>    common engine behaviors before finalizing. The current leaning is:
>>>>       1. Allow all parameters to be nullable.
>>>>       2. Allow the return type to include an explicit nullability
>>>>       annotation.
>>>>
>>>> You can watch the recording here for more details:
>>>> https://youtu.be/IUd-SN0CQRs
>>>> Will update the PR(https://github.com/apache/iceberg/pull/14117) soon.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>>>
>>>>>    1. Replace “definitions version” with “definitions logs”, which
>>>>>    record overload changes and timestamps for informational purposes only.
>>>>>    2. Use a globally increasing numeric ID for overloads instead of
>>>>>    UUIDs.
>>>>>    3. Add explicit types to all fields in the spec to clarify the
>>>>>    metadata structure.
>>>>>    4. Add null-handling optimization hints to help engines skip
>>>>>    evaluation when inputs are null. (Thanks, Talat!)
>>>>>    5. Function parameters remain ordered, but we don’t need to
>>>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>>    nested structs.
>>>>>
>>>>> You can watch the recording here for more details:
>>>>> https://youtu.be/gJnueQODWvs
>>>>> I have updated the PR (https://github.com/apache/iceberg/pull/14117)
>>>>> for the above feedback. Please take a look and let me know if you have any
>>>>> questions.
>>>>>
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>>>
>>>>>>    1. Instead of relying on dynamic inference, the return table’s
>>>>>>    schema for user-defined table functions (UDTFs) should be explicitly
>>>>>>    defined.
>>>>>>    2. Whether a function is a UDTF should be captured as a dedicated
>>>>>>    attribute, rather than being inferred indirectly from the return type.
>>>>>>    3. The interpretation of a UDF body (whether it is treated as a
>>>>>>    partial SQL expression or as a full SELECT statement) should be determined
>>>>>>    by engines. Example: `SELECT x +1` vs. `x + 1`. Different engines have
>>>>>>    different takes.
>>>>>>    4. User-defined aggregation functions (UDAFs) are out of scope
>>>>>>    for now.
>>>>>>    5. Each overload should include its own current-version field.
>>>>>>    This avoids relying solely on the global `definition-versions` when
>>>>>>    querying the current version of one overload.
>>>>>>
>>>>>> You can watch the recording here:
>>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>>> shortly.
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Really appreciated feedback from you all over the past few months.
>>>>>>> I've filed the initial PR for the UDF spec:
>>>>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>>>>> consensus we've built and addresses the write amplification concern raised
>>>>>>> in our last discussion.
>>>>>>>
>>>>>>> Please take a look and share your thoughts. Happy to discuss it
>>>>>>> further during Monday's meeting as well.
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>>
>>>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>> .
>>>>>>>>
>>>>>>>> We also discussed a way to avoid copying every overload into the
>>>>>>>> new metadata JSON when creating a new version. One of ideas is to introduce
>>>>>>>> a global version array, this is not yet reflected in the doc, but I’ll
>>>>>>>> update it shortly. Other key points:
>>>>>>>>
>>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>>    execute.
>>>>>>>>    - Keeping the version while referring to an UDF probably isn't
>>>>>>>>    a good idea. Users are responsible for updating downstream views if they
>>>>>>>>    reference older UDF versions.
>>>>>>>>
>>>>>>>> You can watch the recording here:
>>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>>>>>>>> discussed the UDF metadata structure, captured at this doc(
>>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>>> ). Here is the detailed summary:
>>>>>>>>>
>>>>>>>>>    1. Each UDF overload has its own return type. e.g., `add(int,
>>>>>>>>>    int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>>    2. Return type should be explicitly specified, no implicit or
>>>>>>>>>    statement-based return type inference should be allowed.
>>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>>    properties at the overload level.
>>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>>    5. Introducing a dedicated signature definitions section to
>>>>>>>>>    centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>>    6. Whether to have versioned open properties or not. Versioned
>>>>>>>>>    properties can lead to unnecessary copying of a bag of properties into each
>>>>>>>>>    version, while it provides a clear history of properties for any future
>>>>>>>>>    debugging and understanding of the UDF behavior at a specific point in
>>>>>>>>>    time.
>>>>>>>>>
>>>>>>>>> Watch the recording here,
>>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>>>>> Apologies for the delay!
>>>>>>>>>>
>>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>>       - We agreed to combine overloads with the same name into a
>>>>>>>>>>       single UDF entity, which shares a common metadata.json file.
>>>>>>>>>>       - Listing UDFs will return a list of UDF names, not a list
>>>>>>>>>>       of individual signatures.
>>>>>>>>>>       - Loading a UDF by name will return all of its overloads.
>>>>>>>>>>    - Versioning Strategy
>>>>>>>>>>       - A global version number will track changes across the
>>>>>>>>>>       entire UDF entity, it increments monolithically.
>>>>>>>>>>       - Each overload will also maintain its own version (e.g.,
>>>>>>>>>>       updated_at_version) to trace changes specific to that overload.
>>>>>>>>>>    - For simplicity, the load API will not support
>>>>>>>>>>    argument-based filtering in the initial release. It will always return all
>>>>>>>>>>    overloads for a given UDF name, overload-level loading is not supported at
>>>>>>>>>>    this stage.
>>>>>>>>>>
>>>>>>>>>> Watch the recording here,
>>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> To recap and add my thoughts, we want to support UDFs with
>>>>>>>>>>> multiple signatures under the same name, which can serve both
>>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>>
>>>>>>>>>>> Per my investigation[1], most engines support overloading by
>>>>>>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>>>>>>> change. Here is an example:
>>>>>>>>>>>
>>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>>    - Engine’s best-match resolution now binds the same call to
>>>>>>>>>>>    the new overload, changing behavior without modifying the query.
>>>>>>>>>>>
>>>>>>>>>>> To mitigate this issue, we have to choose between these two
>>>>>>>>>>> access control models:
>>>>>>>>>>>
>>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads
>>>>>>>>>>>    of a function name.
>>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>>>>>>>    signatures.
>>>>>>>>>>>
>>>>>>>>>>> The general recommendation is to adopt *Model A.* It trades
>>>>>>>>>>> some precision for safety and simplicity, while eliminating the silent
>>>>>>>>>>> behavior change problem. More details are in this doc[1].
>>>>>>>>>>>
>>>>>>>>>>> 1.
>>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>>
>>>>>>>>>>>> Summary:
>>>>>>>>>>>> We have discussed how to define function identifiers (should
>>>>>>>>>>>> also handle function overloading). Ryan suggested that we should check how
>>>>>>>>>>>> Spark does it. We can refer to functions using an identifier and then bind
>>>>>>>>>>>> the different signatures to it. So that access policies can be applied per
>>>>>>>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>>>>>>>> when overloading is supported.
>>>>>>>>>>>>
>>>>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>>>>
>>>>>>>>>>>> Please check/subscribe to the dev events calendar for the next
>>>>>>>>>>>> meeting link (Aug 11).
>>>>>>>>>>>>
>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev
>>>>>>>>>>>>> Events" calendar for tomorrow 7/28 at 9AM PT. I missed the
>>>>>>>>>>>>> last one, but i'll be at this one.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No one joined the sync today. I came to know that Yufei is on
>>>>>>>>>>>>>> holiday, and Ryan and others couldn't make it, similar to the last sync. It
>>>>>>>>>>>>>> seems Yufei might have forgotten to transfer meeting ownership as well, as
>>>>>>>>>>>>>> new members needed admin approval and couldn't join automatically this
>>>>>>>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've updated the function signature schema and other open
>>>>>>>>>>>>>> points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I’d propose to move the field `properties` from a top level
>>>>>>>>>>>>>>> field to a field inside “version” along with a representation, so that
>>>>>>>>>>>>>>> properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    1. Each version still enforces a single signature.
>>>>>>>>>>>>>>>>    Although the signatures array is global to the UDF, each version references
>>>>>>>>>>>>>>>>    just one signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they should
>>>>>>>>>>>>>>>>    remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the
>>>>>>>>>>>>>>>>>    engines and in the roadmaps of many engines. Iceberg will support it. We
>>>>>>>>>>>>>>>>>    will maintain the `FunctionIdentifier` (extends `TableIdentifer` but also
>>>>>>>>>>>>>>>>>    have a member containing the function argument's type list). And all
>>>>>>>>>>>>>>>>>    operations like load, rename, list, create and drop are based on
>>>>>>>>>>>>>>>>>    `FunctionIdentifier`.
>>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag,
>>>>>>>>>>>>>>>>>    we need to standardize the property name. Iceberg encryption may be
>>>>>>>>>>>>>>>>>    orthogonal to this discussion.
>>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Standardizing the property names (deterministic,
>>>>>>>>>>>>>>>>>    secure).
>>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>>    - Replace function. To check upto what level replace
>>>>>>>>>>>>>>>>>    is supported (considering function overloading) .
>>>>>>>>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    I think we are close on the spec. Please review the
>>>>>>>>>>>>>>>>>    proposal
>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole
>>>>>>>>>>>>>>>>>> metadata is encrypted, we don't have to worry about just hiding the UDF
>>>>>>>>>>>>>>>>>> body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the spec
>>>>>>>>>>>>>>>>>>>> would require? Would it require hiding the UDF definition from users and
>>>>>>>>>>>>>>>>>>>> require specific pushdown cases be disabled? The use cases seem valid, but
>>>>>>>>>>>>>>>>>>>> I'm trying to understand the requirements this places on engines and why it
>>>>>>>>>>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing
>>>>>>>>>>>>>>>>>>>>>    the UDF body and details like the list of imports, some of them aren’t
>>>>>>>>>>>>>>>>>>>>>    applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an
>>>>>>>>>>>>>>>>>>>>>    isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For
>>>>>>>>>>>>>>>>>>>>>    example, secure UDFs may disable certain optimizations—such as predicate
>>>>>>>>>>>>>>>>>>>>>    pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that
>>>>>>>>>>>>>>>>>>>>> the secure flag is primarily an instruction to the
>>>>>>>>>>>>>>>>>>>>> engine to behave differently. While it's largely an engine-side behavior,
>>>>>>>>>>>>>>>>>>>>> we still need to include this flag in the UDF definition to indicate
>>>>>>>>>>>>>>>>>>>>> whether a UDF is secure, especially considering the perf penalty introduced
>>>>>>>>>>>>>>>>>>>>> by scenario #3. We should clearly recommend that users avoid marking UDFs
>>>>>>>>>>>>>>>>>>>>> as secure unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a
>>>>>>>>>>>>>>>>>>>>>> "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted engine".
>>>>>>>>>>>>>>>>>>>>>> When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but
>>>>>>>>>>>>>>>>>>>>>>> whether those statements run within a single transaction should be treated
>>>>>>>>>>>>>>>>>>>>>>> as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure
>>>>>>>>>>>>>>>>>>>>>>> UDF” option supported by some engines[1], so the body and any sensitive
>>>>>>>>>>>>>>>>>>>>>>> details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator,
>>>>>>>>>>>>>>>>>>>>>>>>    like $$ or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to
>>>>>>>>>>>>>>>>>>>>>>>>    check whether these engines support function overloading for SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>    Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal document
>>>>>>>>>>>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it
>>>>>>>>>>>>>>>>>>>>>>>> is ready.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the
>>>>>>>>>>>>>>>>>>>>>>>>>    majority agreed *not to* (see recording for
>>>>>>>>>>>>>>>>>>>>>>>>>    details).
>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will
>>>>>>>>>>>>>>>>>>>>>>>>>    be included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each
>>>>>>>>>>>>>>>>>>>>>>>>>    catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will remain
>>>>>>>>>>>>>>>>>>>>>>>>>    *required* for cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once
>>>>>>>>>>>>>>>>>>>>>>>>>    it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync
>>>>>>>>>>>>>>>>>>>>>>>>>> for the UDF project. Everyone’s welcome to drop in and share ideas! Here is
>>>>>>>>>>>>>>>>>>>>>>>>>> the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to
>>>>>>>>>>>>>>>>>>>>>>>>>>> discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need
>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning for UDFs at this stage? We explored the possibility of
>>>>>>>>>>>>>>>>>>>>>>>>>>> simplifying the specification by avoiding view replication, and potentially
>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing versioning support later. UDTFs, being a superset of views in
>>>>>>>>>>>>>>>>>>>>>>>>>>> some ways, may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines
>>>>>>>>>>>>>>>>>>>>>>>>>>> may not support vararg syntax in CREATE FUNCTION,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs could represent such arguments as lists when supported by the
>>>>>>>>>>>>>>>>>>>>>>>>>>> engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently
>>>>>>>>>>>>>>>>>>>>>>>>>>> doesn’t support generic types (e.g., object),
>>>>>>>>>>>>>>>>>>>>>>>>>>> we can only map engine-specific types to Iceberg types. As a result,
>>>>>>>>>>>>>>>>>>>>>>>>>>> generic data types will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a
>>>>>>>>>>>>>>>>>>>>>>>>>>> language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two
>>>>>>>>>>>>>>>>>>>>>>>>>>> primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>> examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to
>>>>>>>>>>>>>>>>>>>>>>>>>>> more interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the previous comments. Additionally, I've included the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported by various vendors, including Dremio, Snowflake, Databricks, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> deeper discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was significant interest in storing UDFs in Iceberg and adding endpoints
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this proposal focuses on SQL-based engines, while Python-based systems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> often work with data frames. Adding imperative languages like Python would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fully understand yet how the proposed design addresses shareability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> written in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and added comments. I think it would be helpful to also have a design doc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures is limiting. It would be nice to see a discussion of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the trade-offs and a rationale for the choice. I think it would also be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very helpful to have a few representative use cases for this included in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc. That way the proposal can demonstrate that it solves those use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is inconsistent with conventions in other specs. For example, using string
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec versioning than the Iceberg view and table specs. It requires readers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to fail if there are any unknown fields, which prevents the spec from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding things that are fully backward-compatible. Other Iceberg specs only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> require a version change to introduce forward-incompatible changes and I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think that this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple function signatures per verison, but it is unclear how to encode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating functions across engines, so this doesn’t show that the metadata
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function shows a SELECT statement and it isn’t clear how this is distinct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might be interested. Resharing since I do not think it was directly linked
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a variation of (1) where the API is data flow/data pipeline API instead
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of SQL (e.g., Spark Scala). Yes, that is also possible in the very long run
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some long term opportunities in this case. Consider you register a Spark
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temp view as some sort of data frame read, then it could still be resolved
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to a Spark plan that is representable by an intermediate representation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But I agree this gets very complicated very soon, and just having the case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:40 PM Benny Chow <bt...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disconnect about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha's references are pretty much from (1) and I think those have more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> analogy to views due to their SQL nature. Agree (2) is not practical to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintain by Iceberg, but I think Ajantha's use cases are around (1), and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports SQL representations of UDFs (similar to views as shared by the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reference links above), the complexity involved will be similar to managing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:33 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to have a common set of functions across engines, I don't see how that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the views? I feel we can say exactly the same thing for Iceberg views, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet we have Iceberg multi-dialect views implemented. Maybe it sounds like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support is a good idea, even just a multi-dialect one like view, and that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:32 AM Robert Stupp <sn...@snazy.de>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> common base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a negative touch to the idea of having UDFs in Iceberg, especially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether it's a good idea to add UDFs tracked by Iceberg catalogs. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that Iceberg primarily deals with things that are centralized, like tables
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gauge the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec addition for storing the versioned UDFs in Iceberg (inspired by view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines. Potentially engines can understand the UDFs written by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> begun drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

Thanks for joining today’s community UDF sync! Here’s a quick recap of what
we discussed and agreed on:
1. We’ll keep parameters only at the definition layer to support
named-argument invocation without extra work from the engine side.
2. We agreed to add support for default parameter values.
3. Moving forward with a flag for nullable vs. non-nullable return values,
treating it as a hint rather than a strict rule.
4. The udf metadata will preserve the original case of parameter names, and
engines can decide whether to normalize or convert them.

You can watch the recording here for more details:
https://www.youtube.com/watch?v=0i6nKfDNdGw
Sorry that today's sync wasn't in the Iceberg Dev Event calendar(It was
removed somehow). I have added it back to Iceberg Dev Event as a biweekly
sync. Let me know if you have any questions.

I will update the PR(https://github.com/apache/iceberg/pull/14117) for item
1 and 2 soon. The overall content will remain largely the same. Please take
a look when you get a chance!

Yufei


On Mon, Nov 3, 2025 at 9:05 AM Yufei Gu <fl...@gmail.com> wrote:

> Ryan and me are in this meeting, please join
>
> Iceberg UDF sync
> Monday, November 3 · 9:00 – 10:00am
> Time zone: America/Los_Angeles
> Google Meet joining info
> Video call link: https://meet.google.com/siq-jvtk-tdo
>
> Yufei
>
>
> On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks a lot for the review! Made a few changes to the spec:
>> 1. Renamed "overload" to "definition"
>> 2. Changed definition id from long to string with the format of parameter
>> type tuple.
>> 3. Explicitly claimed that named argument invocation is supported
>> 4. Added a nullable flag for return values
>> 5. Added a few rules for inputs and returns
>>
>> Please find more details in the PR(
>> https://github.com/apache/iceberg/pull/14117). Here is a doc comparing
>> engine behaviors in terms of named argument invocation, nullable return
>> value, default values for inputs, null handling:
>> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
>> This can help for decision making.
>>
>> We could discuss the following items in Monday's meeting:
>> 1. Whether we need to push down parameters to representation given we
>> want to support named argument invocation, it requires extra mapping on the
>> engine side.
>> 2. Whether we support default values.  Spark plans to support it,
>> Snowflake supports it already.
>>
>>
>> Yufei
>>
>>
>> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Thanks for joining today’s community UDF sync. Here’s a summary of what
>>> we discussed and agreed upon:
>>>
>>>    1. We agreed to use Iceberg types for each field in the UDF spec.
>>>    This approach ensures consistency with the table spec and allows for
>>>    straightforward conversion to JSON types.
>>>    2. Using numeric overload IDs can have side effects, for instance,
>>>    deleting an overload and later creating a new one with the same signature
>>>    will use a different numeric id, which points to the same overload. The
>>>    consensus is to represent the overload ID as a string derived from
>>>    parameter types and their order.
>>>    3. Parameter names must remain consistent across all overload
>>>    versions. We agreed to add explicit rules:
>>>       1. Parameter names must match across all overload versions.
>>>       2. Renaming a parameter requires removing all previous versions
>>>       that use the old name.
>>>    4. We discussed whether parameters or return types can be nullable
>>>    and how engines should handle null values. We’ll research SQL standards and
>>>    common engine behaviors before finalizing. The current leaning is:
>>>       1. Allow all parameters to be nullable.
>>>       2. Allow the return type to include an explicit nullability
>>>       annotation.
>>>
>>> You can watch the recording here for more details:
>>> https://youtu.be/IUd-SN0CQRs
>>> Will update the PR(https://github.com/apache/iceberg/pull/14117) soon.
>>>
>>> Yufei
>>>
>>>
>>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>>
>>>>    1. Replace “definitions version” with “definitions logs”, which
>>>>    record overload changes and timestamps for informational purposes only.
>>>>    2. Use a globally increasing numeric ID for overloads instead of
>>>>    UUIDs.
>>>>    3. Add explicit types to all fields in the spec to clarify the
>>>>    metadata structure.
>>>>    4. Add null-handling optimization hints to help engines skip
>>>>    evaluation when inputs are null. (Thanks, Talat!)
>>>>    5. Function parameters remain ordered, but we don’t need to
>>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>>    implementation to ensure consistent behavior across engines, especially for
>>>>    nested structs.
>>>>
>>>> You can watch the recording here for more details:
>>>> https://youtu.be/gJnueQODWvs
>>>> I have updated the PR (https://github.com/apache/iceberg/pull/14117)
>>>> for the above feedback. Please take a look and let me know if you have any
>>>> questions.
>>>>
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>>
>>>>>    1. Instead of relying on dynamic inference, the return table’s
>>>>>    schema for user-defined table functions (UDTFs) should be explicitly
>>>>>    defined.
>>>>>    2. Whether a function is a UDTF should be captured as a dedicated
>>>>>    attribute, rather than being inferred indirectly from the return type.
>>>>>    3. The interpretation of a UDF body (whether it is treated as a
>>>>>    partial SQL expression or as a full SELECT statement) should be determined
>>>>>    by engines. Example: `SELECT x +1` vs. `x + 1`. Different engines have
>>>>>    different takes.
>>>>>    4. User-defined aggregation functions (UDAFs) are out of scope for
>>>>>    now.
>>>>>    5. Each overload should include its own current-version field.
>>>>>    This avoids relying solely on the global `definition-versions` when
>>>>>    querying the current version of one overload.
>>>>>
>>>>> You can watch the recording here:
>>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>>> shortly.
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> Really appreciated feedback from you all over the past few months.
>>>>>> I've filed the initial PR for the UDF spec:
>>>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>>>> consensus we've built and addresses the write amplification concern raised
>>>>>> in our last discussion.
>>>>>>
>>>>>> Please take a look and share your thoughts. Happy to discuss it
>>>>>> further during Monday's meeting as well.
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>>
>>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>> .
>>>>>>>
>>>>>>> We also discussed a way to avoid copying every overload into the new
>>>>>>> metadata JSON when creating a new version. One of ideas is to introduce a
>>>>>>> global version array, this is not yet reflected in the doc, but I’ll update
>>>>>>> it shortly. Other key points:
>>>>>>>
>>>>>>>    - The latest UDF version will typically be used in most
>>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>>    execute.
>>>>>>>    - Keeping the version while referring to an UDF probably isn't a
>>>>>>>    good idea. Users are responsible for updating downstream views if they
>>>>>>>    reference older UDF versions.
>>>>>>>
>>>>>>> You can watch the recording here:
>>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>>>>>>> discussed the UDF metadata structure, captured at this doc(
>>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>>> ). Here is the detailed summary:
>>>>>>>>
>>>>>>>>    1. Each UDF overload has its own return type. e.g., `add(int,
>>>>>>>>    int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>>    2. Return type should be explicitly specified, no implicit or
>>>>>>>>    statement-based return type inference should be allowed.
>>>>>>>>    3. Adding explicit properties like deterministic, doc
>>>>>>>>    properties at the overload level.
>>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>>    5. Introducing a dedicated signature definitions section to
>>>>>>>>    centralize metadata (Function parameters, Return type, Parameter
>>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>>    6. Whether to have versioned open properties or not. Versioned
>>>>>>>>    properties can lead to unnecessary copying of a bag of properties into each
>>>>>>>>    version, while it provides a clear history of properties for any future
>>>>>>>>    debugging and understanding of the UDF behavior at a specific point in
>>>>>>>>    time.
>>>>>>>>
>>>>>>>> Watch the recording here,
>>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>>>> Apologies for the delay!
>>>>>>>>>
>>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>>       - We agreed to combine overloads with the same name into a
>>>>>>>>>       single UDF entity, which shares a common metadata.json file.
>>>>>>>>>       - Listing UDFs will return a list of UDF names, not a list
>>>>>>>>>       of individual signatures.
>>>>>>>>>       - Loading a UDF by name will return all of its overloads.
>>>>>>>>>    - Versioning Strategy
>>>>>>>>>       - A global version number will track changes across the
>>>>>>>>>       entire UDF entity, it increments monolithically.
>>>>>>>>>       - Each overload will also maintain its own version (e.g.,
>>>>>>>>>       updated_at_version) to trace changes specific to that overload.
>>>>>>>>>    - For simplicity, the load API will not support argument-based
>>>>>>>>>    filtering in the initial release. It will always return all overloads for a
>>>>>>>>>    given UDF name, overload-level loading is not supported at this stage.
>>>>>>>>>
>>>>>>>>> Watch the recording here,
>>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> To recap and add my thoughts, we want to support UDFs with
>>>>>>>>>> multiple signatures under the same name, which can serve both
>>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>>
>>>>>>>>>> Per my investigation[1], most engines support overloading by
>>>>>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>>>>>> change. Here is an example:
>>>>>>>>>>
>>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>>    - Engine’s best-match resolution now binds the same call to
>>>>>>>>>>    the new overload, changing behavior without modifying the query.
>>>>>>>>>>
>>>>>>>>>> To mitigate this issue, we have to choose between these two
>>>>>>>>>> access control models:
>>>>>>>>>>
>>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads of
>>>>>>>>>>    a function name.
>>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>>>>>>    signatures.
>>>>>>>>>>
>>>>>>>>>> The general recommendation is to adopt *Model A.* It trades some
>>>>>>>>>> precision for safety and simplicity, while eliminating the silent behavior
>>>>>>>>>> change problem. More details are in this doc[1].
>>>>>>>>>>
>>>>>>>>>> 1.
>>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>>
>>>>>>>>>>> Summary:
>>>>>>>>>>> We have discussed how to define function identifiers (should
>>>>>>>>>>> also handle function overloading). Ryan suggested that we should check how
>>>>>>>>>>> Spark does it. We can refer to functions using an identifier and then bind
>>>>>>>>>>> the different signatures to it. So that access policies can be applied per
>>>>>>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>>>>>>> when overloading is supported.
>>>>>>>>>>>
>>>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>>>
>>>>>>>>>>> Please check/subscribe to the dev events calendar for the next
>>>>>>>>>>> meeting link (Aug 11).
>>>>>>>>>>>
>>>>>>>>>>> - Ajantha
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <
>>>>>>>>>>> kevinjqliu@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>>
>>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev
>>>>>>>>>>>> Events" calendar for tomorrow 7/28 at 9AM PT. I missed the
>>>>>>>>>>>> last one, but i'll be at this one.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> No one joined the sync today. I came to know that Yufei is on
>>>>>>>>>>>>> holiday, and Ryan and others couldn't make it, similar to the last sync. It
>>>>>>>>>>>>> seems Yufei might have forgotten to transfer meeting ownership as well, as
>>>>>>>>>>>>> new members needed admin approval and couldn't join automatically this
>>>>>>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've updated the function signature schema and other open
>>>>>>>>>>>>> points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I’d propose to move the field `properties` from a top level
>>>>>>>>>>>>>> field to a field inside “version” along with a representation, so that
>>>>>>>>>>>>>> properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    1. Each version still enforces a single signature.
>>>>>>>>>>>>>>>    Although the signatures array is global to the UDF, each version references
>>>>>>>>>>>>>>>    just one signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing
>>>>>>>>>>>>>>>    component (signatures) from the more dynamic one (representations) to
>>>>>>>>>>>>>>>    reduce metadata file size.
>>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they should
>>>>>>>>>>>>>>>    remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the engines
>>>>>>>>>>>>>>>>    and in the roadmaps of many engines. Iceberg will support it. We will
>>>>>>>>>>>>>>>>    maintain the `FunctionIdentifier` (extends `TableIdentifer` but also have a
>>>>>>>>>>>>>>>>    member containing the function argument's type list). And all operations
>>>>>>>>>>>>>>>>    like load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag, we
>>>>>>>>>>>>>>>>    need to standardize the property name. Iceberg encryption may be orthogonal
>>>>>>>>>>>>>>>>    to this discussion.
>>>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Standardizing the property names (deterministic,
>>>>>>>>>>>>>>>>    secure).
>>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>>>>>>>>>>    supported (considering function overloading) .
>>>>>>>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    I think we are close on the spec. Please review the
>>>>>>>>>>>>>>>>    proposal
>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole
>>>>>>>>>>>>>>>>> metadata is encrypted, we don't have to worry about just hiding the UDF
>>>>>>>>>>>>>>>>> body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the spec
>>>>>>>>>>>>>>>>>>> would require? Would it require hiding the UDF definition from users and
>>>>>>>>>>>>>>>>>>> require specific pushdown cases be disabled? The use cases seem valid, but
>>>>>>>>>>>>>>>>>>> I'm trying to understand the requirements this places on engines and why it
>>>>>>>>>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing
>>>>>>>>>>>>>>>>>>>>    the UDF body and details like the list of imports, some of them aren’t
>>>>>>>>>>>>>>>>>>>>    applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an
>>>>>>>>>>>>>>>>>>>>    isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For
>>>>>>>>>>>>>>>>>>>>    example, secure UDFs may disable certain optimizations—such as predicate
>>>>>>>>>>>>>>>>>>>>    pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that the
>>>>>>>>>>>>>>>>>>>> secure flag is primarily an instruction to the engine
>>>>>>>>>>>>>>>>>>>> to behave differently. While it's largely an engine-side behavior, we still
>>>>>>>>>>>>>>>>>>>> need to include this flag in the UDF definition to indicate whether a UDF
>>>>>>>>>>>>>>>>>>>> is secure, especially considering the perf penalty introduced by scenario
>>>>>>>>>>>>>>>>>>>> #3. We should clearly recommend that users avoid marking UDFs as secure
>>>>>>>>>>>>>>>>>>>> unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a
>>>>>>>>>>>>>>>>>>>>> "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted engine".
>>>>>>>>>>>>>>>>>>>>> When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but
>>>>>>>>>>>>>>>>>>>>>> whether those statements run within a single transaction should be treated
>>>>>>>>>>>>>>>>>>>>>> as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure
>>>>>>>>>>>>>>>>>>>>>> UDF” option supported by some engines[1], so the body and any sensitive
>>>>>>>>>>>>>>>>>>>>>> details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator, like
>>>>>>>>>>>>>>>>>>>>>>>    $$ or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to
>>>>>>>>>>>>>>>>>>>>>>>    check whether these engines support function overloading for SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>>    Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal document
>>>>>>>>>>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>>>>>>>> ready.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the
>>>>>>>>>>>>>>>>>>>>>>>>    majority agreed *not to* (see recording for
>>>>>>>>>>>>>>>>>>>>>>>>    details).
>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be
>>>>>>>>>>>>>>>>>>>>>>>>    included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each
>>>>>>>>>>>>>>>>>>>>>>>>    catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will remain
>>>>>>>>>>>>>>>>>>>>>>>>    *required* for cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once
>>>>>>>>>>>>>>>>>>>>>>>>    it is ready.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync
>>>>>>>>>>>>>>>>>>>>>>>>> for the UDF project. Everyone’s welcome to drop in and share ideas! Here is
>>>>>>>>>>>>>>>>>>>>>>>>> the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to
>>>>>>>>>>>>>>>>>>>>>>>>>> discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning
>>>>>>>>>>>>>>>>>>>>>>>>>> for UDFs at this stage? We explored the possibility of simplifying the
>>>>>>>>>>>>>>>>>>>>>>>>>> specification by avoiding view replication, and potentially introducing
>>>>>>>>>>>>>>>>>>>>>>>>>> versioning support later. UDTFs, being a superset of views in some ways,
>>>>>>>>>>>>>>>>>>>>>>>>>> may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines
>>>>>>>>>>>>>>>>>>>>>>>>>> may not support vararg syntax in CREATE FUNCTION,
>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs could represent such arguments as lists when supported by the
>>>>>>>>>>>>>>>>>>>>>>>>>> engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently
>>>>>>>>>>>>>>>>>>>>>>>>>> doesn’t support generic types (e.g., object), we
>>>>>>>>>>>>>>>>>>>>>>>>>> can only map engine-specific types to Iceberg types. As a result, generic
>>>>>>>>>>>>>>>>>>>>>>>>>> data types will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a
>>>>>>>>>>>>>>>>>>>>>>>>>> language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two
>>>>>>>>>>>>>>>>>>>>>>>>>> primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax
>>>>>>>>>>>>>>>>>>>>>>>>>> examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to
>>>>>>>>>>>>>>>>>>>>>>>>>> more interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the
>>>>>>>>>>>>>>>>>>>>>>>>>>> previous comments. Additionally, I've included the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>>>>>>>>> by various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a
>>>>>>>>>>>>>>>>>>>>>>>>>>> deeper discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there
>>>>>>>>>>>>>>>>>>>>>>>>>>>> was significant interest in storing UDFs in Iceberg and adding endpoints
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> support, I’d like to take ownership of this effort and revive the work.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'll be revisiting the document I proposed long back and will share an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this proposal focuses on SQL-based engines, while Python-based systems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> often work with data frames. Adding imperative languages like Python would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> added comments. I think it would be helpful to also have a design doc that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures is limiting. It would be nice to see a discussion of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the trade-offs and a rationale for the choice. I think it would also be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very helpful to have a few representative use cases for this included in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc. That way the proposal can demonstrate that it solves those use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is inconsistent with conventions in other specs. For example, using string
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IDs rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec versioning than the Iceberg view and table specs. It requires readers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to fail if there are any unknown fields, which prevents the spec from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding things that are fully backward-compatible. Other Iceberg specs only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> require a version change to introduce forward-incompatible changes and I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think that this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple function signatures per verison, but it is unclear how to encode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating functions across engines, so this doesn’t show that the metadata
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function shows a SELECT statement and it isn’t clear how this is distinct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might be interested. Resharing since I do not think it was directly linked
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> take the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some long term opportunities in this case. Consider you register a Spark
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temp view as some sort of data frame read, then it could still be resolved
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to a Spark plan that is representable by an intermediate representation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But I agree this gets very complicated very soon, and just having the case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benny Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defined by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports SQL representations of UDFs (similar to views as shared by the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reference links above), the complexity involved will be similar to managing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:33 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to have a common set of functions across engines, I don't see how that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the views? I feel we can say exactly the same thing for Iceberg views, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet we have Iceberg multi-dialect views implemented. Maybe it sounds like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support is a good idea, even just a multi-dialect one like view, and that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:32 AM Robert Stupp <sn...@snazy.de>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific and portable and "non-centralized" as views are. The same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> common base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a negative touch to the idea of having UDFs in Iceberg, especially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not in this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whether it's a good idea to add UDFs tracked by Iceberg catalogs. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that Iceberg primarily deals with things that are centralized, like tables
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gauge the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec addition for storing the versioned UDFs in Iceberg (inspired by view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between the engines. Potentially engines can understand the UDFs written by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Ryan and me are in this meeting, please join

Iceberg UDF sync
Monday, November 3 · 9:00 – 10:00am
Time zone: America/Los_Angeles
Google Meet joining info
Video call link: https://meet.google.com/siq-jvtk-tdo

Yufei


On Sun, Nov 2, 2025 at 1:48 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> Thanks a lot for the review! Made a few changes to the spec:
> 1. Renamed "overload" to "definition"
> 2. Changed definition id from long to string with the format of parameter
> type tuple.
> 3. Explicitly claimed that named argument invocation is supported
> 4. Added a nullable flag for return values
> 5. Added a few rules for inputs and returns
>
> Please find more details in the PR(
> https://github.com/apache/iceberg/pull/14117). Here is a doc comparing
> engine behaviors in terms of named argument invocation, nullable return
> value, default values for inputs, null handling:
> https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
> This can help for decision making.
>
> We could discuss the following items in Monday's meeting:
> 1. Whether we need to push down parameters to representation given we want
> to support named argument invocation, it requires extra mapping on the
> engine side.
> 2. Whether we support default values.  Spark plans to support it,
> Snowflake supports it already.
>
>
> Yufei
>
>
> On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks for joining today’s community UDF sync. Here’s a summary of what
>> we discussed and agreed upon:
>>
>>    1. We agreed to use Iceberg types for each field in the UDF spec.
>>    This approach ensures consistency with the table spec and allows for
>>    straightforward conversion to JSON types.
>>    2. Using numeric overload IDs can have side effects, for instance,
>>    deleting an overload and later creating a new one with the same signature
>>    will use a different numeric id, which points to the same overload. The
>>    consensus is to represent the overload ID as a string derived from
>>    parameter types and their order.
>>    3. Parameter names must remain consistent across all overload
>>    versions. We agreed to add explicit rules:
>>       1. Parameter names must match across all overload versions.
>>       2. Renaming a parameter requires removing all previous versions
>>       that use the old name.
>>    4. We discussed whether parameters or return types can be nullable
>>    and how engines should handle null values. We’ll research SQL standards and
>>    common engine behaviors before finalizing. The current leaning is:
>>       1. Allow all parameters to be nullable.
>>       2. Allow the return type to include an explicit nullability
>>       annotation.
>>
>> You can watch the recording here for more details:
>> https://youtu.be/IUd-SN0CQRs
>> Will update the PR(https://github.com/apache/iceberg/pull/14117) soon.
>>
>> Yufei
>>
>>
>> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>>
>>>    1. Replace “definitions version” with “definitions logs”, which
>>>    record overload changes and timestamps for informational purposes only.
>>>    2. Use a globally increasing numeric ID for overloads instead of
>>>    UUIDs.
>>>    3. Add explicit types to all fields in the spec to clarify the
>>>    metadata structure.
>>>    4. Add null-handling optimization hints to help engines skip
>>>    evaluation when inputs are null. (Thanks, Talat!)
>>>    5. Function parameters remain ordered, but we don’t need to
>>>    introduce IDs for each parameter. We’ll double-check the Java
>>>    implementation to ensure consistent behavior across engines, especially for
>>>    nested structs.
>>>
>>> You can watch the recording here for more details:
>>> https://youtu.be/gJnueQODWvs
>>> I have updated the PR (https://github.com/apache/iceberg/pull/14117)
>>> for the above feedback. Please take a look and let me know if you have any
>>> questions.
>>>
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>>
>>>>    1. Instead of relying on dynamic inference, the return table’s
>>>>    schema for user-defined table functions (UDTFs) should be explicitly
>>>>    defined.
>>>>    2. Whether a function is a UDTF should be captured as a dedicated
>>>>    attribute, rather than being inferred indirectly from the return type.
>>>>    3. The interpretation of a UDF body (whether it is treated as a
>>>>    partial SQL expression or as a full SELECT statement) should be determined
>>>>    by engines. Example: `SELECT x +1` vs. `x + 1`. Different engines have
>>>>    different takes.
>>>>    4. User-defined aggregation functions (UDAFs) are out of scope for
>>>>    now.
>>>>    5. Each overload should include its own current-version field. This
>>>>    avoids relying solely on the global `definition-versions` when querying the
>>>>    current version of one overload.
>>>>
>>>> You can watch the recording here:
>>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>>> shortly.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Really appreciated feedback from you all over the past few months.
>>>>> I've filed the initial PR for the UDF spec:
>>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>>> consensus we've built and addresses the write amplification concern raised
>>>>> in our last discussion.
>>>>>
>>>>> Please take a look and share your thoughts. Happy to discuss it
>>>>> further during Monday's meeting as well.
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>>
>>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>> .
>>>>>>
>>>>>> We also discussed a way to avoid copying every overload into the new
>>>>>> metadata JSON when creating a new version. One of ideas is to introduce a
>>>>>> global version array, this is not yet reflected in the doc, but I’ll update
>>>>>> it shortly. Other key points:
>>>>>>
>>>>>>    - The latest UDF version will typically be used in most
>>>>>>    scenarios, but engines retain the flexibility to choose which version to
>>>>>>    execute.
>>>>>>    - Keeping the version while referring to an UDF probably isn't a
>>>>>>    good idea. Users are responsible for updating downstream views if they
>>>>>>    reference older UDF versions.
>>>>>>
>>>>>> You can watch the recording here:
>>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>>>>>> discussed the UDF metadata structure, captured at this doc(
>>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>>> ). Here is the detailed summary:
>>>>>>>
>>>>>>>    1. Each UDF overload has its own return type. e.g., `add(int,
>>>>>>>    int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>>    2. Return type should be explicitly specified, no implicit or
>>>>>>>    statement-based return type inference should be allowed.
>>>>>>>    3. Adding explicit properties like deterministic, doc properties
>>>>>>>    at the overload level.
>>>>>>>    4. Adding property “secure” at the top level.
>>>>>>>    5. Introducing a dedicated signature definitions section to
>>>>>>>    centralize metadata (Function parameters, Return type, Parameter
>>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>>    6. Whether to have versioned open properties or not. Versioned
>>>>>>>    properties can lead to unnecessary copying of a bag of properties into each
>>>>>>>    version, while it provides a clear history of properties for any future
>>>>>>>    debugging and understanding of the UDF behavior at a specific point in
>>>>>>>    time.
>>>>>>>
>>>>>>> Watch the recording here,
>>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>>> Apologies for the delay!
>>>>>>>>
>>>>>>>>    - One UDF entity for all overloads
>>>>>>>>       - We agreed to combine overloads with the same name into a
>>>>>>>>       single UDF entity, which shares a common metadata.json file.
>>>>>>>>       - Listing UDFs will return a list of UDF names, not a list
>>>>>>>>       of individual signatures.
>>>>>>>>       - Loading a UDF by name will return all of its overloads.
>>>>>>>>    - Versioning Strategy
>>>>>>>>       - A global version number will track changes across the
>>>>>>>>       entire UDF entity, it increments monolithically.
>>>>>>>>       - Each overload will also maintain its own version (e.g.,
>>>>>>>>       updated_at_version) to trace changes specific to that overload.
>>>>>>>>    - For simplicity, the load API will not support argument-based
>>>>>>>>    filtering in the initial release. It will always return all overloads for a
>>>>>>>>    given UDF name, overload-level loading is not supported at this stage.
>>>>>>>>
>>>>>>>> Watch the recording here,
>>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> To recap and add my thoughts, we want to support UDFs with
>>>>>>>>> multiple signatures under the same name, which can serve both
>>>>>>>>> overload-aware and overload-naive engines.
>>>>>>>>>
>>>>>>>>> Per my investigation[1], most engines support overloading by
>>>>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>>>>> change. Here is an example:
>>>>>>>>>
>>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>>    - Engine’s best-match resolution now binds the same call to
>>>>>>>>>    the new overload, changing behavior without modifying the query.
>>>>>>>>>
>>>>>>>>> To mitigate this issue, we have to choose between these two access
>>>>>>>>> control models:
>>>>>>>>>
>>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads of
>>>>>>>>>    a function name.
>>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>>>>>    signatures.
>>>>>>>>>
>>>>>>>>> The general recommendation is to adopt *Model A.* It trades some
>>>>>>>>> precision for safety and simplicity, while eliminating the silent behavior
>>>>>>>>> change problem. More details are in this doc[1].
>>>>>>>>>
>>>>>>>>> 1.
>>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>>
>>>>>>>>>> Summary:
>>>>>>>>>> We have discussed how to define function identifiers (should also
>>>>>>>>>> handle function overloading). Ryan suggested that we should check how Spark
>>>>>>>>>> does it. We can refer to functions using an identifier and then bind the
>>>>>>>>>> different signatures to it. So that access policies can be applied per
>>>>>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>>>>>> when overloading is supported.
>>>>>>>>>>
>>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>>
>>>>>>>>>> Please check/subscribe to the dev events calendar for the next
>>>>>>>>>> meeting link (Aug 11).
>>>>>>>>>>
>>>>>>>>>> - Ajantha
>>>>>>>>>>
>>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>>
>>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev
>>>>>>>>>>> Events" calendar for tomorrow 7/28 at 9AM PT. I missed the last
>>>>>>>>>>> one, but i'll be at this one.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Kevin Liu
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> No one joined the sync today. I came to know that Yufei is on
>>>>>>>>>>>> holiday, and Ryan and others couldn't make it, similar to the last sync. It
>>>>>>>>>>>> seems Yufei might have forgotten to transfer meeting ownership as well, as
>>>>>>>>>>>> new members needed admin approval and couldn't join automatically this
>>>>>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>>
>>>>>>>>>>>> I've updated the function signature schema and other open
>>>>>>>>>>>> points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>>
>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I’d propose to move the field `properties` from a top level
>>>>>>>>>>>>> field to a field inside “version” along with a representation, so that
>>>>>>>>>>>>> properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    1. Each version still enforces a single signature.
>>>>>>>>>>>>>>    Although the signatures array is global to the UDF, each version references
>>>>>>>>>>>>>>    just one signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>>>    2. We’ve separated the less frequently changing component
>>>>>>>>>>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>>>>>>>>>>    file size.
>>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they should
>>>>>>>>>>>>>>    remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Function overloading: Supported by few of the engines
>>>>>>>>>>>>>>>    and in the roadmaps of many engines. Iceberg will support it. We will
>>>>>>>>>>>>>>>    maintain the `FunctionIdentifier` (extends `TableIdentifer` but also have a
>>>>>>>>>>>>>>>    member containing the function argument's type list). And all operations
>>>>>>>>>>>>>>>    like load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag, we
>>>>>>>>>>>>>>>    need to standardize the property name. Iceberg encryption may be orthogonal
>>>>>>>>>>>>>>>    to this discussion.
>>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Standardizing the property names (deterministic,
>>>>>>>>>>>>>>>    secure).
>>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>>>>>>>>>    supported (considering function overloading) .
>>>>>>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    I think we are close on the spec. Please review the
>>>>>>>>>>>>>>>    proposal
>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole
>>>>>>>>>>>>>>>> metadata is encrypted, we don't have to worry about just hiding the UDF
>>>>>>>>>>>>>>>> body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <
>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the spec
>>>>>>>>>>>>>>>>>> would require? Would it require hiding the UDF definition from users and
>>>>>>>>>>>>>>>>>> require specific pushdown cases be disabled? The use cases seem valid, but
>>>>>>>>>>>>>>>>>> I'm trying to understand the requirements this places on engines and why it
>>>>>>>>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the
>>>>>>>>>>>>>>>>>>>    UDF body and details like the list of imports, some of them aren’t
>>>>>>>>>>>>>>>>>>>    applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an
>>>>>>>>>>>>>>>>>>>    isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For
>>>>>>>>>>>>>>>>>>>    example, secure UDFs may disable certain optimizations—such as predicate
>>>>>>>>>>>>>>>>>>>    pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that the
>>>>>>>>>>>>>>>>>>> secure flag is primarily an instruction to the engine
>>>>>>>>>>>>>>>>>>> to behave differently. While it's largely an engine-side behavior, we still
>>>>>>>>>>>>>>>>>>> need to include this flag in the UDF definition to indicate whether a UDF
>>>>>>>>>>>>>>>>>>> is secure, especially considering the perf penalty introduced by scenario
>>>>>>>>>>>>>>>>>>> #3. We should clearly recommend that users avoid marking UDFs as secure
>>>>>>>>>>>>>>>>>>> unless it's truly necessary.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a
>>>>>>>>>>>>>>>>>>>> "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted engine".
>>>>>>>>>>>>>>>>>>>> When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but
>>>>>>>>>>>>>>>>>>>>> whether those statements run within a single transaction should be treated
>>>>>>>>>>>>>>>>>>>>> as an engine-level concern. The Iceberg UDF spec can spell out the
>>>>>>>>>>>>>>>>>>>>> expectation, yet the actual guarantee still depends on the runtime. Even if
>>>>>>>>>>>>>>>>>>>>> a UDF declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure
>>>>>>>>>>>>>>>>>>>>> UDF” option supported by some engines[1], so the body and any sensitive
>>>>>>>>>>>>>>>>>>>>> details stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator, like
>>>>>>>>>>>>>>>>>>>>>>    $$ or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to
>>>>>>>>>>>>>>>>>>>>>>    check whether these engines support function overloading for SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>>    Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal document
>>>>>>>>>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>>>>>>> ready.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the
>>>>>>>>>>>>>>>>>>>>>>>    majority agreed *not to* (see recording for
>>>>>>>>>>>>>>>>>>>>>>>    details).
>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be
>>>>>>>>>>>>>>>>>>>>>>>    included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each
>>>>>>>>>>>>>>>>>>>>>>>    catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will remain
>>>>>>>>>>>>>>>>>>>>>>>    *required* for cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once
>>>>>>>>>>>>>>>>>>>>>>>    it is ready.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync
>>>>>>>>>>>>>>>>>>>>>>>> for the UDF project. Everyone’s welcome to drop in and share ideas! Here is
>>>>>>>>>>>>>>>>>>>>>>>> the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to
>>>>>>>>>>>>>>>>>>>>>>>>> discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning
>>>>>>>>>>>>>>>>>>>>>>>>> for UDFs at this stage? We explored the possibility of simplifying the
>>>>>>>>>>>>>>>>>>>>>>>>> specification by avoiding view replication, and potentially introducing
>>>>>>>>>>>>>>>>>>>>>>>>> versioning support later. UDTFs, being a superset of views in some ways,
>>>>>>>>>>>>>>>>>>>>>>>>> may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines
>>>>>>>>>>>>>>>>>>>>>>>>> may not support vararg syntax in CREATE FUNCTION,
>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs could represent such arguments as lists when supported by the
>>>>>>>>>>>>>>>>>>>>>>>>> engine.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently
>>>>>>>>>>>>>>>>>>>>>>>>> doesn’t support generic types (e.g., object), we
>>>>>>>>>>>>>>>>>>>>>>>>> can only map engine-specific types to Iceberg types. As a result, generic
>>>>>>>>>>>>>>>>>>>>>>>>> data types will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a
>>>>>>>>>>>>>>>>>>>>>>>>> language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two
>>>>>>>>>>>>>>>>>>>>>>>>> primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax
>>>>>>>>>>>>>>>>>>>>>>>>> examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the
>>>>>>>>>>>>>>>>>>>>>>>>>> previous comments. Additionally, I've included the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>>>>>>>> by various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there
>>>>>>>>>>>>>>>>>>>>>>>>>>> was significant interest in storing UDFs in Iceberg and adding endpoints
>>>>>>>>>>>>>>>>>>>>>>>>>>> for UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and support,
>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to take ownership of this effort and revive the work. I'll be
>>>>>>>>>>>>>>>>>>>>>>>>>>> revisiting the document I proposed long back and will share an updated
>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only SQL representations, although it is certainly faviouring SQL in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples... It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this proposal focuses on SQL-based engines, while Python-based systems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> often work with data frames. Adding imperative languages like Python would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> added comments. I think it would be helpful to also have a design doc that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures is limiting. It would be nice to see a discussion of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the trade-offs and a rationale for the choice. I think it would also be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very helpful to have a few representative use cases for this included in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc. That way the proposal can demonstrate that it solves those use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec versioning than the Iceberg view and table specs. It requires readers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to fail if there are any unknown fields, which prevents the spec from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding things that are fully backward-compatible. Other Iceberg specs only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> require a version change to introduce forward-incompatible changes and I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think that this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple function signatures per verison, but it is unclear how to encode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating functions across engines, so this doesn’t show that the metadata
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shows a SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might be interested. Resharing since I do not think it was directly linked
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some long term opportunities in this case. Consider you register a Spark
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temp view as some sort of data frame read, then it could still be resolved
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to a Spark plan that is representable by an intermediate representation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But I agree this gets very complicated very soon, and just having the case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benny Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports SQL representations of UDFs (similar to views as shared by the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reference links above), the complexity involved will be similar to managing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:33 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a common set of functions across engines, I don't see how that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the views? I feel we can say exactly the same thing for Iceberg views, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet we have Iceberg multi-dialect views implemented. Maybe it sounds like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support is a good idea, even just a multi-dialect one like view, and that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:32 AM Robert Stupp <sn...@snazy.de>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and portable and "non-centralized" as views are. The same performance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> common base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's a good idea to add UDFs tracked by Iceberg catalogs. I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg primarily deals with things that are centralized, like tables of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when you post the proposal, but I think this would be a very difficult area
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to tackle across engines, languages, and memory models without having a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> huge performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gauge the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec addition for storing the versioned UDFs in Iceberg (inspired by view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integrating this feature into Iceberg would be a valuable addition, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we're eager to collaborate with the community to develop a UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

Thanks a lot for the review! Made a few changes to the spec:
1. Renamed "overload" to "definition"
2. Changed definition id from long to string with the format of parameter
type tuple.
3. Explicitly claimed that named argument invocation is supported
4. Added a nullable flag for return values
5. Added a few rules for inputs and returns

Please find more details in the PR(
https://github.com/apache/iceberg/pull/14117). Here is a doc comparing
engine behaviors in terms of named argument invocation, nullable return
value, default values for inputs, null handling:
https://docs.google.com/document/d/1GC896Z4gxYP0Vz-ENqZ3tZZBqXEUQf4qDJO11NRo8F4/edit?usp=sharing.
This can help for decision making.

We could discuss the following items in Monday's meeting:
1. Whether we need to push down parameters to representation given we want
to support named argument invocation, it requires extra mapping on the
engine side.
2. Whether we support default values.  Spark plans to support it, Snowflake
supports it already.


Yufei


On Mon, Oct 20, 2025 at 4:15 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> Thanks for joining today’s community UDF sync. Here’s a summary of what we
> discussed and agreed upon:
>
>    1. We agreed to use Iceberg types for each field in the UDF spec. This
>    approach ensures consistency with the table spec and allows for
>    straightforward conversion to JSON types.
>    2. Using numeric overload IDs can have side effects, for instance,
>    deleting an overload and later creating a new one with the same signature
>    will use a different numeric id, which points to the same overload. The
>    consensus is to represent the overload ID as a string derived from
>    parameter types and their order.
>    3. Parameter names must remain consistent across all overload
>    versions. We agreed to add explicit rules:
>       1. Parameter names must match across all overload versions.
>       2. Renaming a parameter requires removing all previous versions
>       that use the old name.
>    4. We discussed whether parameters or return types can be nullable and
>    how engines should handle null values. We’ll research SQL standards and
>    common engine behaviors before finalizing. The current leaning is:
>       1. Allow all parameters to be nullable.
>       2. Allow the return type to include an explicit nullability
>       annotation.
>
> You can watch the recording here for more details:
> https://youtu.be/IUd-SN0CQRs
> Will update the PR(https://github.com/apache/iceberg/pull/14117) soon.
>
> Yufei
>
>
> On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>>
>>    1. Replace “definitions version” with “definitions logs”, which
>>    record overload changes and timestamps for informational purposes only.
>>    2. Use a globally increasing numeric ID for overloads instead of
>>    UUIDs.
>>    3. Add explicit types to all fields in the spec to clarify the
>>    metadata structure.
>>    4. Add null-handling optimization hints to help engines skip
>>    evaluation when inputs are null. (Thanks, Talat!)
>>    5. Function parameters remain ordered, but we don’t need to introduce
>>    IDs for each parameter. We’ll double-check the Java implementation to
>>    ensure consistent behavior across engines, especially for nested structs.
>>
>> You can watch the recording here for more details:
>> https://youtu.be/gJnueQODWvs
>> I have updated the PR (https://github.com/apache/iceberg/pull/14117) for
>> the above feedback. Please take a look and let me know if you have any
>> questions.
>>
>>
>> Yufei
>>
>>
>> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>>
>>>    1. Instead of relying on dynamic inference, the return table’s
>>>    schema for user-defined table functions (UDTFs) should be explicitly
>>>    defined.
>>>    2. Whether a function is a UDTF should be captured as a dedicated
>>>    attribute, rather than being inferred indirectly from the return type.
>>>    3. The interpretation of a UDF body (whether it is treated as a
>>>    partial SQL expression or as a full SELECT statement) should be determined
>>>    by engines. Example: `SELECT x +1` vs. `x + 1`. Different engines have
>>>    different takes.
>>>    4. User-defined aggregation functions (UDAFs) are out of scope for
>>>    now.
>>>    5. Each overload should include its own current-version field. This
>>>    avoids relying solely on the global `definition-versions` when querying the
>>>    current version of one overload.
>>>
>>> You can watch the recording here:
>>> https://www.youtube.com/watch?v=9t2xev8WfAw
>>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>>> shortly.
>>>
>>> Yufei
>>>
>>>
>>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Really appreciated feedback from you all over the past few months. I've
>>>> filed the initial PR for the UDF spec:
>>>> https://github.com/apache/iceberg/pull/14117. It captures the
>>>> consensus we've built and addresses the write amplification concern raised
>>>> in our last discussion.
>>>>
>>>> Please take a look and share your thoughts. Happy to discuss it further
>>>> during Monday's meeting as well.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>>
>>>>> We covered the UDF metadata structure, captured in this doc:
>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>> .
>>>>>
>>>>> We also discussed a way to avoid copying every overload into the new
>>>>> metadata JSON when creating a new version. One of ideas is to introduce a
>>>>> global version array, this is not yet reflected in the doc, but I’ll update
>>>>> it shortly. Other key points:
>>>>>
>>>>>    - The latest UDF version will typically be used in most scenarios,
>>>>>    but engines retain the flexibility to choose which version to execute.
>>>>>    - Keeping the version while referring to an UDF probably isn't a
>>>>>    good idea. Users are responsible for updating downstream views if they
>>>>>    reference older UDF versions.
>>>>>
>>>>> You can watch the recording here:
>>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>>>>> discussed the UDF metadata structure, captured at this doc(
>>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>>> ). Here is the detailed summary:
>>>>>>
>>>>>>    1. Each UDF overload has its own return type. e.g., `add(int,
>>>>>>    int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>>    2. Return type should be explicitly specified, no implicit or
>>>>>>    statement-based return type inference should be allowed.
>>>>>>    3. Adding explicit properties like deterministic, doc properties
>>>>>>    at the overload level.
>>>>>>    4. Adding property “secure” at the top level.
>>>>>>    5. Introducing a dedicated signature definitions section to
>>>>>>    centralize metadata (Function parameters, Return type, Parameter
>>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>>    a table schema doesn’t create a new snapshot.
>>>>>>    6. Whether to have versioned open properties or not. Versioned
>>>>>>    properties can lead to unnecessary copying of a bag of properties into each
>>>>>>    version, while it provides a clear history of properties for any future
>>>>>>    debugging and understanding of the UDF behavior at a specific point in
>>>>>>    time.
>>>>>>
>>>>>> Watch the recording here,
>>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone, here’s the summary from our last sync on 8/11.
>>>>>>> Apologies for the delay!
>>>>>>>
>>>>>>>    - One UDF entity for all overloads
>>>>>>>       - We agreed to combine overloads with the same name into a
>>>>>>>       single UDF entity, which shares a common metadata.json file.
>>>>>>>       - Listing UDFs will return a list of UDF names, not a list of
>>>>>>>       individual signatures.
>>>>>>>       - Loading a UDF by name will return all of its overloads.
>>>>>>>    - Versioning Strategy
>>>>>>>       - A global version number will track changes across the
>>>>>>>       entire UDF entity, it increments monolithically.
>>>>>>>       - Each overload will also maintain its own version (e.g.,
>>>>>>>       updated_at_version) to trace changes specific to that overload.
>>>>>>>    - For simplicity, the load API will not support argument-based
>>>>>>>    filtering in the initial release. It will always return all overloads for a
>>>>>>>    given UDF name, overload-level loading is not supported at this stage.
>>>>>>>
>>>>>>> Watch the recording here,
>>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> To recap and add my thoughts, we want to support UDFs with multiple
>>>>>>>> signatures under the same name, which can serve both overload-aware and
>>>>>>>> overload-naive engines.
>>>>>>>>
>>>>>>>> Per my investigation[1], most engines support overloading by
>>>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>>>> change. Here is an example:
>>>>>>>>
>>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>>    - Engine’s best-match resolution now binds the same call to the
>>>>>>>>    new overload, changing behavior without modifying the query.
>>>>>>>>
>>>>>>>> To mitigate this issue, we have to choose between these two access
>>>>>>>> control models:
>>>>>>>>
>>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads of a
>>>>>>>>    function name.
>>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>>>>    signatures.
>>>>>>>>
>>>>>>>> The general recommendation is to adopt *Model A.* It trades some
>>>>>>>> precision for safety and simplicity, while eliminating the silent behavior
>>>>>>>> change problem. More details are in this doc[1].
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <aj...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>> Here is the meeting recording:
>>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>>
>>>>>>>>> Summary:
>>>>>>>>> We have discussed how to define function identifiers (should also
>>>>>>>>> handle function overloading). Ryan suggested that we should check how Spark
>>>>>>>>> does it. We can refer to functions using an identifier and then bind the
>>>>>>>>> different signatures to it. So that access policies can be applied per
>>>>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>>>>> when overloading is supported.
>>>>>>>>>
>>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>>
>>>>>>>>> Please check/subscribe to the dev events calendar for the next
>>>>>>>>> meeting link (Aug 11).
>>>>>>>>>
>>>>>>>>> - Ajantha
>>>>>>>>>
>>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Ajantha,
>>>>>>>>>>
>>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev Events"
>>>>>>>>>> calendar for tomorrow 7/28 at 9AM PT. I missed the last one, but
>>>>>>>>>> i'll be at this one.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Kevin Liu
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>
>>>>>>>>>>> No one joined the sync today. I came to know that Yufei is on
>>>>>>>>>>> holiday, and Ryan and others couldn't make it, similar to the last sync. It
>>>>>>>>>>> seems Yufei might have forgotten to transfer meeting ownership as well, as
>>>>>>>>>>> new members needed admin approval and couldn't join automatically this
>>>>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>>
>>>>>>>>>>> I've updated the function signature schema and other open
>>>>>>>>>>> points. I believe we're very close to the final version of the spec. A
>>>>>>>>>>> meeting is indeed necessary to finalize this, but we don't have to wait for
>>>>>>>>>>> it to finish the review process. We had many meetings on this in the past
>>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>>
>>>>>>>>>>> - Ajantha
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I’d propose to move the field `properties` from a top level
>>>>>>>>>>>> field to a field inside “version” along with a representation, so that
>>>>>>>>>>>> properties are versioned. A property like “deterministic” could change
>>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>>> won't be safe.
>>>>>>>>>>>>
>>>>>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1. Each version still enforces a single signature.
>>>>>>>>>>>>>    Although the signatures array is global to the UDF, each version references
>>>>>>>>>>>>>    just one signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>>    2. We’ve separated the less frequently changing component
>>>>>>>>>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>>>>>>>>>    file size.
>>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they should
>>>>>>>>>>>>>    remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Function overloading: Supported by few of the engines
>>>>>>>>>>>>>>    and in the roadmaps of many engines. Iceberg will support it. We will
>>>>>>>>>>>>>>    maintain the `FunctionIdentifier` (extends `TableIdentifer` but also have a
>>>>>>>>>>>>>>    member containing the function argument's type list). And all operations
>>>>>>>>>>>>>>    like load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag, we
>>>>>>>>>>>>>>    need to standardize the property name. Iceberg encryption may be orthogonal
>>>>>>>>>>>>>>    to this discussion.
>>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Standardizing the property names (deterministic,
>>>>>>>>>>>>>>    secure).
>>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>>>>>>>>    supported (considering function overloading) .
>>>>>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    I think we are close on the spec. Please review the
>>>>>>>>>>>>>>    proposal
>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>>    .
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole
>>>>>>>>>>>>>>> metadata is encrypted, we don't have to worry about just hiding the UDF
>>>>>>>>>>>>>>> body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the spec
>>>>>>>>>>>>>>>>> would require? Would it require hiding the UDF definition from users and
>>>>>>>>>>>>>>>>> require specific pushdown cases be disabled? The use cases seem valid, but
>>>>>>>>>>>>>>>>> I'm trying to understand the requirements this places on engines and why it
>>>>>>>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the
>>>>>>>>>>>>>>>>>>    UDF body and details like the list of imports, some of them aren’t
>>>>>>>>>>>>>>>>>>    applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an
>>>>>>>>>>>>>>>>>>    isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For
>>>>>>>>>>>>>>>>>>    example, secure UDFs may disable certain optimizations—such as predicate
>>>>>>>>>>>>>>>>>>    pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that the
>>>>>>>>>>>>>>>>>> secure flag is primarily an instruction to the engine to
>>>>>>>>>>>>>>>>>> behave differently. While it's largely an engine-side behavior, we still
>>>>>>>>>>>>>>>>>> need to include this flag in the UDF definition to indicate whether a UDF
>>>>>>>>>>>>>>>>>> is secure, especially considering the perf penalty introduced by scenario
>>>>>>>>>>>>>>>>>> #3. We should clearly recommend that users avoid marking UDFs as secure
>>>>>>>>>>>>>>>>>> unless it's truly necessary.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a
>>>>>>>>>>>>>>>>>>> "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted engine".
>>>>>>>>>>>>>>>>>>> When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether
>>>>>>>>>>>>>>>>>>>> those statements run within a single transaction should be treated as an
>>>>>>>>>>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure UDF”
>>>>>>>>>>>>>>>>>>>> option supported by some engines[1], so the body and any sensitive details
>>>>>>>>>>>>>>>>>>>> stay hidden from callers?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax
>>>>>>>>>>>>>>>>>>>>>    supported by different engines (Snowflake, databricks, Dremio, Trino, OSS
>>>>>>>>>>>>>>>>>>>>>    spark 4.0).
>>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator, like
>>>>>>>>>>>>>>>>>>>>>    $$ or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to
>>>>>>>>>>>>>>>>>>>>>    check whether these engines support function overloading for SQL UDFs.
>>>>>>>>>>>>>>>>>>>>>    Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>>>>>> ready.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the
>>>>>>>>>>>>>>>>>>>>>>    majority agreed *not to* (see recording for
>>>>>>>>>>>>>>>>>>>>>>    details).
>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be
>>>>>>>>>>>>>>>>>>>>>>    included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each
>>>>>>>>>>>>>>>>>>>>>>    catalog resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will remain
>>>>>>>>>>>>>>>>>>>>>>    *required* for cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once it
>>>>>>>>>>>>>>>>>>>>>>    is ready.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync
>>>>>>>>>>>>>>>>>>>>>>> for the UDF project. Everyone’s welcome to drop in and share ideas! Here is
>>>>>>>>>>>>>>>>>>>>>>> the meeting link:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>>> Video call link:
>>>>>>>>>>>>>>>>>>>>>>> https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to
>>>>>>>>>>>>>>>>>>>>>>>> discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning
>>>>>>>>>>>>>>>>>>>>>>>> for UDFs at this stage? We explored the possibility of simplifying the
>>>>>>>>>>>>>>>>>>>>>>>> specification by avoiding view replication, and potentially introducing
>>>>>>>>>>>>>>>>>>>>>>>> versioning support later. UDTFs, being a superset of views in some ways,
>>>>>>>>>>>>>>>>>>>>>>>> may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may
>>>>>>>>>>>>>>>>>>>>>>>> not support vararg syntax in CREATE FUNCTION,
>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs could represent such arguments as lists when supported by the
>>>>>>>>>>>>>>>>>>>>>>>> engine.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently
>>>>>>>>>>>>>>>>>>>>>>>> doesn’t support generic types (e.g., object), we
>>>>>>>>>>>>>>>>>>>>>>>> can only map engine-specific types to Iceberg types. As a result, generic
>>>>>>>>>>>>>>>>>>>>>>>> data types will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a
>>>>>>>>>>>>>>>>>>>>>>>> language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two
>>>>>>>>>>>>>>>>>>>>>>>> primary UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax
>>>>>>>>>>>>>>>>>>>>>>>> examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the
>>>>>>>>>>>>>>>>>>>>>>>>> previous comments. Additionally, I've included the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>>>>>>> by various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and support,
>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to take ownership of this effort and revive the work. I'll be
>>>>>>>>>>>>>>>>>>>>>>>>>> revisiting the document I proposed long back and will share an updated
>>>>>>>>>>>>>>>>>>>>>>>>>> proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to
>>>>>>>>>>>>>>>>>>>>>>>>>>> be SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new
>>>>>>>>>>>>>>>>>>>>>>>>>>> type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the spec. It leaves a way for future versions to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>> different representations later, but only SQL is supported. That was also
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the feedback to my initial skepticism about how it would work to add
>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL representations, although it is certainly faviouring SQL in examples...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Driesprong <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this proposal focuses on SQL-based engines, while Python-based systems
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> often work with data frames. Adding imperative languages like Python would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hard to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> added comments. I think it would be helpful to also have a design doc that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures is limiting. It would be nice to see a discussion of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the trade-offs and a rationale for the choice. I think it would also be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very helpful to have a few representative use cases for this included in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc. That way the proposal can demonstrate that it solves those use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spec versioning than the Iceberg view and table specs. It requires readers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to fail if there are any unknown fields, which prevents the spec from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding things that are fully backward-compatible. Other Iceberg specs only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> require a version change to introduce forward-incompatible changes and I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> think that this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple function signatures per verison, but it is unclear how to encode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating functions across engines, so this doesn’t show that the metadata
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shows a SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might be interested. Resharing since I do not think it was directly linked
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> far (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some long term opportunities in this case. Consider you register a Spark
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temp view as some sort of data frame read, then it could still be resolved
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to a Spark plan that is representable by an intermediate representation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But I agree this gets very complicated very soon, and just having the case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benny Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports SQL representations of UDFs (similar to views as shared by the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reference links above), the complexity involved will be similar to managing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack, for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 7:33 PM Jack Ye <ye...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a common set of functions across engines, I don't see how that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the views? I feel we can say exactly the same thing for Iceberg views, but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> yet we have Iceberg multi-dialect views implemented. Maybe it sounds like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support is a good idea, even just a multi-dialect one like view, and that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:32 AM Robert Stupp <sn...@snazy.de>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and portable and "non-centralized" as views are. The same performance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> common base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's a good idea to add UDFs tracked by Iceberg catalogs. I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg primarily deals with things that are centralized, like tables of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8:10 AM Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gauge the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> addition for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

Thanks for joining today’s community UDF sync. Here’s a summary of what we
discussed and agreed upon:

   1. We agreed to use Iceberg types for each field in the UDF spec. This
   approach ensures consistency with the table spec and allows for
   straightforward conversion to JSON types.
   2. Using numeric overload IDs can have side effects, for instance,
   deleting an overload and later creating a new one with the same signature
   will use a different numeric id, which points to the same overload. The
   consensus is to represent the overload ID as a string derived from
   parameter types and their order.
   3. Parameter names must remain consistent across all overload versions.
   We agreed to add explicit rules:
      1. Parameter names must match across all overload versions.
      2. Renaming a parameter requires removing all previous versions that
      use the old name.
   4. We discussed whether parameters or return types can be nullable and
   how engines should handle null values. We’ll research SQL standards and
   common engine behaviors before finalizing. The current leaning is:
      1. Allow all parameters to be nullable.
      2. Allow the return type to include an explicit nullability
      annotation.

You can watch the recording here for more details:
https://youtu.be/IUd-SN0CQRs
Will update the PR(https://github.com/apache/iceberg/pull/14117) soon.

Yufei


On Tue, Oct 7, 2025 at 9:25 AM Yufei Gu <fl...@gmail.com> wrote:

> Hi everyone,
>
> Thanks a lot for joining Monday's UDF sync. Here is the summary:
>
>    1. Replace “definitions version” with “definitions logs”, which record
>    overload changes and timestamps for informational purposes only.
>    2. Use a globally increasing numeric ID for overloads instead of UUIDs.
>    3. Add explicit types to all fields in the spec to clarify the
>    metadata structure.
>    4. Add null-handling optimization hints to help engines skip
>    evaluation when inputs are null. (Thanks, Talat!)
>    5. Function parameters remain ordered, but we don’t need to introduce
>    IDs for each parameter. We’ll double-check the Java implementation to
>    ensure consistent behavior across engines, especially for nested structs.
>
> You can watch the recording here for more details:
> https://youtu.be/gJnueQODWvs
> I have updated the PR (https://github.com/apache/iceberg/pull/14117) for
> the above feedback. Please take a look and let me know if you have any
> questions.
>
>
> Yufei
>
>
> On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Thanks a lot for joining today's UDF sync. Here is the summary:
>>
>>    1. Instead of relying on dynamic inference, the return table’s schema
>>    for user-defined table functions (UDTFs) should be explicitly defined.
>>    2. Whether a function is a UDTF should be captured as a dedicated
>>    attribute, rather than being inferred indirectly from the return type.
>>    3. The interpretation of a UDF body (whether it is treated as a
>>    partial SQL expression or as a full SELECT statement) should be determined
>>    by engines. Example: `SELECT x +1` vs. `x + 1`. Different engines have
>>    different takes.
>>    4. User-defined aggregation functions (UDAFs) are out of scope for
>>    now.
>>    5. Each overload should include its own current-version field. This
>>    avoids relying solely on the global `definition-versions` when querying the
>>    current version of one overload.
>>
>> You can watch the recording here:
>> https://www.youtube.com/watch?v=9t2xev8WfAw
>> I will update the PR(https://github.com/apache/iceberg/pull/14117)
>> shortly.
>>
>> Yufei
>>
>>
>> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Really appreciated feedback from you all over the past few months. I've
>>> filed the initial PR for the UDF spec:
>>> https://github.com/apache/iceberg/pull/14117. It captures the consensus
>>> we've built and addresses the write amplification concern raised in our
>>> last discussion.
>>>
>>> Please take a look and share your thoughts. Happy to discuss it further
>>> during Monday's meeting as well.
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks, thanks for joining today’s UDF sync.
>>>>
>>>> We covered the UDF metadata structure, captured in this doc:
>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>> .
>>>>
>>>> We also discussed a way to avoid copying every overload into the new
>>>> metadata JSON when creating a new version. One of ideas is to introduce a
>>>> global version array, this is not yet reflected in the doc, but I’ll update
>>>> it shortly. Other key points:
>>>>
>>>>    - The latest UDF version will typically be used in most scenarios,
>>>>    but engines retain the flexibility to choose which version to execute.
>>>>    - Keeping the version while referring to an UDF probably isn't a
>>>>    good idea. Users are responsible for updating downstream views if they
>>>>    reference older UDF versions.
>>>>
>>>> You can watch the recording here:
>>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>>>> discussed the UDF metadata structure, captured at this doc(
>>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>>> ). Here is the detailed summary:
>>>>>
>>>>>    1. Each UDF overload has its own return type. e.g., `add(int,
>>>>>    int)` returns `int`, while `add(long, long)`  returns `long`
>>>>>    2. Return type should be explicitly specified, no implicit or
>>>>>    statement-based return type inference should be allowed.
>>>>>    3. Adding explicit properties like deterministic, doc properties
>>>>>    at the overload level.
>>>>>    4. Adding property “secure” at the top level.
>>>>>    5. Introducing a dedicated signature definitions section to
>>>>>    centralize metadata (Function parameters, Return type, Parameter
>>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>>    a table schema doesn’t create a new snapshot.
>>>>>    6. Whether to have versioned open properties or not. Versioned
>>>>>    properties can lead to unnecessary copying of a bag of properties into each
>>>>>    version, while it provides a clear history of properties for any future
>>>>>    debugging and understanding of the UDF behavior at a specific point in
>>>>>    time.
>>>>>
>>>>> Watch the recording here,
>>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi everyone, here’s the summary from our last sync on 8/11. Apologies
>>>>>> for the delay!
>>>>>>
>>>>>>    - One UDF entity for all overloads
>>>>>>       - We agreed to combine overloads with the same name into a
>>>>>>       single UDF entity, which shares a common metadata.json file.
>>>>>>       - Listing UDFs will return a list of UDF names, not a list of
>>>>>>       individual signatures.
>>>>>>       - Loading a UDF by name will return all of its overloads.
>>>>>>    - Versioning Strategy
>>>>>>       - A global version number will track changes across the entire
>>>>>>       UDF entity, it increments monolithically.
>>>>>>       - Each overload will also maintain its own version (e.g.,
>>>>>>       updated_at_version) to trace changes specific to that overload.
>>>>>>    - For simplicity, the load API will not support argument-based
>>>>>>    filtering in the initial release. It will always return all overloads for a
>>>>>>    given UDF name, overload-level loading is not supported at this stage.
>>>>>>
>>>>>> Watch the recording here,
>>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>>
>>>>>>> To recap and add my thoughts, we want to support UDFs with multiple
>>>>>>> signatures under the same name, which can serve both overload-aware and
>>>>>>> overload-naive engines.
>>>>>>>
>>>>>>> Per my investigation[1], most engines support overloading by
>>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>>> change. Here is an example:
>>>>>>>
>>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>>    - Engine’s best-match resolution now binds the same call to the
>>>>>>>    new overload, changing behavior without modifying the query.
>>>>>>>
>>>>>>> To mitigate this issue, we have to choose between these two access
>>>>>>> control models:
>>>>>>>
>>>>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads of a
>>>>>>>    function name.
>>>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>>>    signatures.
>>>>>>>
>>>>>>> The general recommendation is to adopt *Model A.* It trades some
>>>>>>> precision for safety and simplicity, while eliminating the silent behavior
>>>>>>> change problem. More details are in this doc[1].
>>>>>>>
>>>>>>> 1.
>>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>> Here is the meeting recording:
>>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>>
>>>>>>>> Summary:
>>>>>>>> We have discussed how to define function identifiers (should also
>>>>>>>> handle function overloading). Ryan suggested that we should check how Spark
>>>>>>>> does it. We can refer to functions using an identifier and then bind the
>>>>>>>> different signatures to it. So that access policies can be applied per
>>>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>>>> when overloading is supported.
>>>>>>>>
>>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>>
>>>>>>>> Please check/subscribe to the dev events calendar for the next
>>>>>>>> meeting link (Aug 11).
>>>>>>>>
>>>>>>>> - Ajantha
>>>>>>>>
>>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Ajantha,
>>>>>>>>>
>>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev Events"
>>>>>>>>> calendar for tomorrow 7/28 at 9AM PT. I missed the last one, but
>>>>>>>>> i'll be at this one.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Kevin Liu
>>>>>>>>>
>>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hey everyone,
>>>>>>>>>>
>>>>>>>>>> No one joined the sync today. I came to know that Yufei is on
>>>>>>>>>> holiday, and Ryan and others couldn't make it, similar to the last sync. It
>>>>>>>>>> seems Yufei might have forgotten to transfer meeting ownership as well, as
>>>>>>>>>> new members needed admin approval and couldn't join automatically this
>>>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>>
>>>>>>>>>> I've updated the function signature schema and other open points.
>>>>>>>>>> I believe we're very close to the final version of the spec. A meeting is
>>>>>>>>>> indeed necessary to finalize this, but we don't have to wait for it to
>>>>>>>>>> finish the review process. We had many meetings on this in the past
>>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>>
>>>>>>>>>> - Ajantha
>>>>>>>>>>
>>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I’d propose to move the field `properties` from a top level
>>>>>>>>>>> field to a field inside “version” along with a representation, so that
>>>>>>>>>>> properties are versioned. A property like “deterministic” could change
>>>>>>>>>>> along with representation over time. For example, we need to change
>>>>>>>>>>> “deterministic” from true to false in case of adding a non-deterministic
>>>>>>>>>>> SQL expression/function(e.g., now()) inside an UDF. Otherwise, rollback
>>>>>>>>>>> won't be safe.
>>>>>>>>>>>
>>>>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>
>>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>>
>>>>>>>>>>>>    1. Each version still enforces a single signature. Although
>>>>>>>>>>>>    the signatures array is global to the UDF, each version references just one
>>>>>>>>>>>>    signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>>    2. We’ve separated the less frequently changing component
>>>>>>>>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>>>>>>>>    file size.
>>>>>>>>>>>>    3. Since signatures use Iceberg data types, they should
>>>>>>>>>>>>    remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>>
>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Function overloading: Supported by few of the engines
>>>>>>>>>>>>>    and in the roadmaps of many engines. Iceberg will support it. We will
>>>>>>>>>>>>>    maintain the `FunctionIdentifier` (extends `TableIdentifer` but also have a
>>>>>>>>>>>>>    member containing the function argument's type list). And all operations
>>>>>>>>>>>>>    like load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag, we
>>>>>>>>>>>>>    need to standardize the property name. Iceberg encryption may be orthogonal
>>>>>>>>>>>>>    to this discussion.
>>>>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>>>>
>>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Standardizing the property names (deterministic, secure).
>>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>>>>>>>    supported (considering function overloading) .
>>>>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>>>>
>>>>>>>>>>>>>    I think we are close on the spec. Please review the
>>>>>>>>>>>>>    proposal
>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>>    .
>>>>>>>>>>>>>
>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole
>>>>>>>>>>>>>> metadata is encrypted, we don't have to worry about just hiding the UDF
>>>>>>>>>>>>>> body? Let us discuss more on the sync today.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <
>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the spec
>>>>>>>>>>>>>>>> would require? Would it require hiding the UDF definition from users and
>>>>>>>>>>>>>>>> require specific pushdown cases be disabled? The use cases seem valid, but
>>>>>>>>>>>>>>>> I'm trying to understand the requirements this places on engines and why it
>>>>>>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the
>>>>>>>>>>>>>>>>>    UDF body and details like the list of imports, some of them aren’t
>>>>>>>>>>>>>>>>>    applicable to SQL UDFs.
>>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an
>>>>>>>>>>>>>>>>>    isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For
>>>>>>>>>>>>>>>>>    example, secure UDFs may disable certain optimizations—such as predicate
>>>>>>>>>>>>>>>>>    pushdown—to avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that the
>>>>>>>>>>>>>>>>> secure flag is primarily an instruction to the engine to
>>>>>>>>>>>>>>>>> behave differently. While it's largely an engine-side behavior, we still
>>>>>>>>>>>>>>>>> need to include this flag in the UDF definition to indicate whether a UDF
>>>>>>>>>>>>>>>>> is secure, especially considering the perf penalty introduced by scenario
>>>>>>>>>>>>>>>>> #3. We should clearly recommend that users avoid marking UDFs as secure
>>>>>>>>>>>>>>>>> unless it's truly necessary.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a
>>>>>>>>>>>>>>>>>> "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted engine".
>>>>>>>>>>>>>>>>>> When the engine interacts with the catalog it sends authorization
>>>>>>>>>>>>>>>>>> information about itself in addition to the user that it is acting on
>>>>>>>>>>>>>>>>>> behalf of. That way the catalog knows that the secure UDF can be sent to
>>>>>>>>>>>>>>>>>> the engine and won't be shown to the user. The majority of this logic is on
>>>>>>>>>>>>>>>>>> the REST server side, and the only part that is communicated to the client
>>>>>>>>>>>>>>>>>> is the request not to show the UDF to the user, right? In that case should
>>>>>>>>>>>>>>>>>> this be a property rather than part of the definition? Even if we state
>>>>>>>>>>>>>>>>>> that the client "must" suppress the UDF definition, it's really just a
>>>>>>>>>>>>>>>>>> request. Only trusted engines can be passed the UDF definition, so a spec
>>>>>>>>>>>>>>>>>> requirement to suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether
>>>>>>>>>>>>>>>>>>> those statements run within a single transaction should be treated as an
>>>>>>>>>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure UDF”
>>>>>>>>>>>>>>>>>>> option supported by some engines[1], so the body and any sensitive details
>>>>>>>>>>>>>>>>>>> stay hidden from callers?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>>    by different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator, like $$
>>>>>>>>>>>>>>>>>>>>    or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to
>>>>>>>>>>>>>>>>>>>>    check whether these engines support function overloading for SQL UDFs.
>>>>>>>>>>>>>>>>>>>>    Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>>>>> ready.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the
>>>>>>>>>>>>>>>>>>>>>    majority agreed *not to* (see recording for
>>>>>>>>>>>>>>>>>>>>>    details).
>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be
>>>>>>>>>>>>>>>>>>>>>    included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each catalog
>>>>>>>>>>>>>>>>>>>>>    resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will remain
>>>>>>>>>>>>>>>>>>>>>    *required* for cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view
>>>>>>>>>>>>>>>>>>>>>    interoperability as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once it
>>>>>>>>>>>>>>>>>>>>>    is ready.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for
>>>>>>>>>>>>>>>>>>>>>> the UDF project. Everyone’s welcome to drop in and share ideas! Here is the
>>>>>>>>>>>>>>>>>>>>>> meeting link:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to
>>>>>>>>>>>>>>>>>>>>>>> discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning
>>>>>>>>>>>>>>>>>>>>>>> for UDFs at this stage? We explored the possibility of simplifying the
>>>>>>>>>>>>>>>>>>>>>>> specification by avoiding view replication, and potentially introducing
>>>>>>>>>>>>>>>>>>>>>>> versioning support later. UDTFs, being a superset of views in some ways,
>>>>>>>>>>>>>>>>>>>>>>> may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may
>>>>>>>>>>>>>>>>>>>>>>> not support vararg syntax in CREATE FUNCTION,
>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs could represent such arguments as lists when supported by the
>>>>>>>>>>>>>>>>>>>>>>> engine.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently
>>>>>>>>>>>>>>>>>>>>>>> doesn’t support generic types (e.g., object), we
>>>>>>>>>>>>>>>>>>>>>>> can only map engine-specific types to Iceberg types. As a result, generic
>>>>>>>>>>>>>>>>>>>>>>> data types will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a
>>>>>>>>>>>>>>>>>>>>>>> language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two primary
>>>>>>>>>>>>>>>>>>>>>>> UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax
>>>>>>>>>>>>>>>>>>>>>>> examples in both SQL and Python, and detail how each use case is
>>>>>>>>>>>>>>>>>>>>>>> represented in Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the
>>>>>>>>>>>>>>>>>>>>>>>> previous comments. Additionally, I've included the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>>>>>> by various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and support,
>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to take ownership of this effort and revive the work. I'll be
>>>>>>>>>>>>>>>>>>>>>>>>> revisiting the document I proposed long back and will share an updated
>>>>>>>>>>>>>>>>>>>>>>>>> proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to
>>>>>>>>>>>>>>>>>>>>>>>>>> be SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new
>>>>>>>>>>>>>>>>>>>>>>>>>> type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the
>>>>>>>>>>>>>>>>>>>>>>>>>>> spec. It leaves a way for future versions to add different representations
>>>>>>>>>>>>>>>>>>>>>>>>>>> later, but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only
>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL representations, although it is certainly faviouring SQL in examples...
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <fo...@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal focuses on SQL-based engines, while Python-based systems often
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work with data frames. Adding imperative languages like Python would make
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread. The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> added comments. I think it would be helpful to also have a design doc that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures is limiting. It would be nice to see a discussion of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the trade-offs and a rationale for the choice. I think it would also be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very helpful to have a few representative use cases for this included in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc. That way the proposal can demonstrate that it solves those use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple function signatures per verison, but it is unclear how to encode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating functions across engines, so this doesn’t show that the metadata
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shows a SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might be interested. Resharing since I do not think it was directly linked
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some long term opportunities in this case. Consider you register a Spark
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> temp view as some sort of data frame read, then it could still be resolved
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to a Spark plan that is representable by an intermediate representation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But I agree this gets very complicated very soon, and just having the case
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (1) covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benny Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by the user whose definition is a composition of other built-in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions/SQL expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports SQL representations of UDFs (similar to views as shared by the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reference links above), the complexity involved will be similar to managing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a common set of functions across engines, I don't see how that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views? I feel we can say exactly the same thing for Iceberg views, but yet
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we have Iceberg multi-dialect views implemented. Maybe it sounds like we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support is a good idea, even just a multi-dialect one like view, and that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1:32 AM Robert Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and portable and "non-centralized" as views are. The same performance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> common base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's a good idea to add UDFs tracked by Iceberg catalogs. I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg primarily deals with things that are centralized, like tables of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8:10 AM Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gauge the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> addition for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi everyone,

Thanks a lot for joining Monday's UDF sync. Here is the summary:

   1. Replace “definitions version” with “definitions logs”, which record
   overload changes and timestamps for informational purposes only.
   2. Use a globally increasing numeric ID for overloads instead of UUIDs.
   3. Add explicit types to all fields in the spec to clarify the metadata
   structure.
   4. Add null-handling optimization hints to help engines skip evaluation
   when inputs are null. (Thanks, Talat!)
   5. Function parameters remain ordered, but we don’t need to introduce
   IDs for each parameter. We’ll double-check the Java implementation to
   ensure consistent behavior across engines, especially for nested structs.

You can watch the recording here for more details:
https://youtu.be/gJnueQODWvs
I have updated the PR (https://github.com/apache/iceberg/pull/14117) for
the above feedback. Please take a look and let me know if you have any
questions.


Yufei


On Mon, Sep 22, 2025 at 2:10 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> Thanks a lot for joining today's UDF sync. Here is the summary:
>
>    1. Instead of relying on dynamic inference, the return table’s schema
>    for user-defined table functions (UDTFs) should be explicitly defined.
>    2. Whether a function is a UDTF should be captured as a dedicated
>    attribute, rather than being inferred indirectly from the return type.
>    3. The interpretation of a UDF body (whether it is treated as a
>    partial SQL expression or as a full SELECT statement) should be determined
>    by engines. Example: `SELECT x +1` vs. `x + 1`. Different engines have
>    different takes.
>    4. User-defined aggregation functions (UDAFs) are out of scope for now.
>    5. Each overload should include its own current-version field. This
>    avoids relying solely on the global `definition-versions` when querying the
>    current version of one overload.
>
> You can watch the recording here:
> https://www.youtube.com/watch?v=9t2xev8WfAw
> I will update the PR(https://github.com/apache/iceberg/pull/14117)
> shortly.
>
> Yufei
>
>
> On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Really appreciated feedback from you all over the past few months. I've
>> filed the initial PR for the UDF spec:
>> https://github.com/apache/iceberg/pull/14117. It captures the consensus
>> we've built and addresses the write amplification concern raised in our
>> last discussion.
>>
>> Please take a look and share your thoughts. Happy to discuss it further
>> during Monday's meeting as well.
>>
>> Yufei
>>
>>
>> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks, thanks for joining today’s UDF sync.
>>>
>>> We covered the UDF metadata structure, captured in this doc:
>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>> .
>>>
>>> We also discussed a way to avoid copying every overload into the new
>>> metadata JSON when creating a new version. One of ideas is to introduce a
>>> global version array, this is not yet reflected in the doc, but I’ll update
>>> it shortly. Other key points:
>>>
>>>    - The latest UDF version will typically be used in most scenarios,
>>>    but engines retain the flexibility to choose which version to execute.
>>>    - Keeping the version while referring to an UDF probably isn't a
>>>    good idea. Users are responsible for updating downstream views if they
>>>    reference older UDF versions.
>>>
>>> You can watch the recording here:
>>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>>> discussed the UDF metadata structure, captured at this doc(
>>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>>> ). Here is the detailed summary:
>>>>
>>>>    1. Each UDF overload has its own return type. e.g., `add(int, int)`
>>>>    returns `int`, while `add(long, long)`  returns `long`
>>>>    2. Return type should be explicitly specified, no implicit or
>>>>    statement-based return type inference should be allowed.
>>>>    3. Adding explicit properties like deterministic, doc properties at
>>>>    the overload level.
>>>>    4. Adding property “secure” at the top level.
>>>>    5. Introducing a dedicated signature definitions section to
>>>>    centralize metadata (Function parameters, Return type, Parameter
>>>>    descriptions). Each overload would reference a signature definition by ID.
>>>>    This decoupling allows signature-related updates (like modifying parameter
>>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>>    a table schema doesn’t create a new snapshot.
>>>>    6. Whether to have versioned open properties or not. Versioned
>>>>    properties can lead to unnecessary copying of a bag of properties into each
>>>>    version, while it provides a clear history of properties for any future
>>>>    debugging and understanding of the UDF behavior at a specific point in
>>>>    time.
>>>>
>>>> Watch the recording here,
>>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi everyone, here’s the summary from our last sync on 8/11. Apologies
>>>>> for the delay!
>>>>>
>>>>>    - One UDF entity for all overloads
>>>>>       - We agreed to combine overloads with the same name into a
>>>>>       single UDF entity, which shares a common metadata.json file.
>>>>>       - Listing UDFs will return a list of UDF names, not a list of
>>>>>       individual signatures.
>>>>>       - Loading a UDF by name will return all of its overloads.
>>>>>    - Versioning Strategy
>>>>>       - A global version number will track changes across the entire
>>>>>       UDF entity, it increments monolithically.
>>>>>       - Each overload will also maintain its own version (e.g.,
>>>>>       updated_at_version) to trace changes specific to that overload.
>>>>>    - For simplicity, the load API will not support argument-based
>>>>>    filtering in the initial release. It will always return all overloads for a
>>>>>    given UDF name, overload-level loading is not supported at this stage.
>>>>>
>>>>> Watch the recording here,
>>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> To recap and add my thoughts, we want to support UDFs with multiple
>>>>>> signatures under the same name, which can serve both overload-aware and
>>>>>> overload-naive engines.
>>>>>>
>>>>>> Per my investigation[1], most engines support overloading by
>>>>>> arguments and allow implicit conversions like numeric widening (e.g., INT →
>>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>>> change. Here is an example:
>>>>>>
>>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>>    - Engine’s best-match resolution now binds the same call to the
>>>>>>    new overload, changing behavior without modifying the query.
>>>>>>
>>>>>> To mitigate this issue, we have to choose between these two access
>>>>>> control models:
>>>>>>
>>>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads of a
>>>>>>    function name.
>>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>>    signatures.
>>>>>>
>>>>>> The general recommendation is to adopt *Model A.* It trades some
>>>>>> precision for safety and simplicity, while eliminating the silent behavior
>>>>>> change problem. More details are in this doc[1].
>>>>>>
>>>>>> 1.
>>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <aj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks to everyone who joined the sync.
>>>>>>> Here is the meeting recording:
>>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>>
>>>>>>> Summary:
>>>>>>> We have discussed how to define function identifiers (should also
>>>>>>> handle function overloading). Ryan suggested that we should check how Spark
>>>>>>> does it. We can refer to functions using an identifier and then bind the
>>>>>>> different signatures to it. So that access policies can be applied per
>>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>>> when overloading is supported.
>>>>>>>
>>>>>>> I will check more about this and update the proposal doc.
>>>>>>>
>>>>>>> Please check/subscribe to the dev events calendar for the next
>>>>>>> meeting link (Aug 11).
>>>>>>>
>>>>>>> - Ajantha
>>>>>>>
>>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Ajantha,
>>>>>>>>
>>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev Events"
>>>>>>>> calendar for tomorrow 7/28 at 9AM PT. I missed the last one, but
>>>>>>>> i'll be at this one.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Kevin Liu
>>>>>>>>
>>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <aj...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hey everyone,
>>>>>>>>>
>>>>>>>>> No one joined the sync today. I came to know that Yufei is on
>>>>>>>>> holiday, and Ryan and others couldn't make it, similar to the last sync. It
>>>>>>>>> seems Yufei might have forgotten to transfer meeting ownership as well, as
>>>>>>>>> new members needed admin approval and couldn't join automatically this
>>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>>
>>>>>>>>> I've updated the function signature schema and other open points.
>>>>>>>>> I believe we're very close to the final version of the spec. A meeting is
>>>>>>>>> indeed necessary to finalize this, but we don't have to wait for it to
>>>>>>>>> finish the review process. We had many meetings on this in the past
>>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>>
>>>>>>>>> - Ajantha
>>>>>>>>>
>>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I’d propose to move the field `properties` from a top level field
>>>>>>>>>> to a field inside “version” along with a representation, so that properties
>>>>>>>>>> are versioned. A property like “deterministic” could change along with
>>>>>>>>>> representation over time. For example, we need to change “deterministic”
>>>>>>>>>> from true to false in case of adding a non-deterministic SQL
>>>>>>>>>> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
>>>>>>>>>> be safe.
>>>>>>>>>>
>>>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>
>>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>>
>>>>>>>>>>>    1. Each version still enforces a single signature. Although
>>>>>>>>>>>    the signatures array is global to the UDF, each version references just one
>>>>>>>>>>>    signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>>    2. We’ve separated the less frequently changing component
>>>>>>>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>>>>>>>    file size.
>>>>>>>>>>>    3. Since signatures use Iceberg data types, they should
>>>>>>>>>>>    remain unaffected by multi-dialect representation differences.
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>>
>>>>>>>>>>>> Summary:
>>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>>
>>>>>>>>>>>>    - Function overloading: Supported by few of the engines and
>>>>>>>>>>>>    in the roadmaps of many engines. Iceberg will support it. We will maintain
>>>>>>>>>>>>    the `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>>>>>>>>>>    containing the function argument's type list). And all operations like
>>>>>>>>>>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag, we
>>>>>>>>>>>>    need to standardize the property name. Iceberg encryption may be orthogonal
>>>>>>>>>>>>    to this discussion.
>>>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>>>
>>>>>>>>>>>> new discussions around
>>>>>>>>>>>>
>>>>>>>>>>>>    - Standardizing the property names (deterministic, secure).
>>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>>>>>>    supported (considering function overloading) .
>>>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>>>
>>>>>>>>>>>>    I think we are close on the spec. Please review the proposal
>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>>    .
>>>>>>>>>>>>
>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>
>>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>
>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole metadata
>>>>>>>>>>>>> is encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>>>>>>>>>>> discuss more on the sync today.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the spec
>>>>>>>>>>>>>>> would require? Would it require hiding the UDF definition from users and
>>>>>>>>>>>>>>> require specific pushdown cases be disabled? The use cases seem valid, but
>>>>>>>>>>>>>>> I'm trying to understand the requirements this places on engines and why it
>>>>>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the
>>>>>>>>>>>>>>>>    UDF body and details like the list of imports, some of them aren’t
>>>>>>>>>>>>>>>>    applicable to SQL UDFs.
>>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an
>>>>>>>>>>>>>>>>    isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For example,
>>>>>>>>>>>>>>>>    secure UDFs may disable certain optimizations—such as predicate pushdown—to
>>>>>>>>>>>>>>>>    avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that the
>>>>>>>>>>>>>>>> secure flag is primarily an instruction to the engine to
>>>>>>>>>>>>>>>> behave differently. While it's largely an engine-side behavior, we still
>>>>>>>>>>>>>>>> need to include this flag in the UDF definition to indicate whether a UDF
>>>>>>>>>>>>>>>> is secure, especially considering the perf penalty introduced by scenario
>>>>>>>>>>>>>>>> #3. We should clearly recommend that users avoid marking UDFs as secure
>>>>>>>>>>>>>>>> unless it's truly necessary.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <
>>>>>>>>>>>>>>>> rdblue@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a
>>>>>>>>>>>>>>>>> "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted engine". When
>>>>>>>>>>>>>>>>> the engine interacts with the catalog it sends authorization information
>>>>>>>>>>>>>>>>> about itself in addition to the user that it is acting on behalf of. That
>>>>>>>>>>>>>>>>> way the catalog knows that the secure UDF can be sent to the engine and
>>>>>>>>>>>>>>>>> won't be shown to the user. The majority of this logic is on the REST
>>>>>>>>>>>>>>>>> server side, and the only part that is communicated to the client is the
>>>>>>>>>>>>>>>>> request not to show the UDF to the user, right? In that case should this be
>>>>>>>>>>>>>>>>> a property rather than part of the definition? Even if we state that the
>>>>>>>>>>>>>>>>> client "must" suppress the UDF definition, it's really just a request. Only
>>>>>>>>>>>>>>>>> trusted engines can be passed the UDF definition, so a spec requirement to
>>>>>>>>>>>>>>>>> suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether
>>>>>>>>>>>>>>>>>> those statements run within a single transaction should be treated as an
>>>>>>>>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure UDF”
>>>>>>>>>>>>>>>>>> option supported by some engines[1], so the body and any sensitive details
>>>>>>>>>>>>>>>>>> stay hidden from callers?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>    by different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator, like $$
>>>>>>>>>>>>>>>>>>>    or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to
>>>>>>>>>>>>>>>>>>>    check whether these engines support function overloading for SQL UDFs.
>>>>>>>>>>>>>>>>>>>    Postgres supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>>>> ready.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the majority
>>>>>>>>>>>>>>>>>>>>    agreed *not to* (see recording for details).
>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be
>>>>>>>>>>>>>>>>>>>>    included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each catalog
>>>>>>>>>>>>>>>>>>>>    resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will remain
>>>>>>>>>>>>>>>>>>>>    *required* for cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view interoperability
>>>>>>>>>>>>>>>>>>>>    as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once it
>>>>>>>>>>>>>>>>>>>>    is ready.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for
>>>>>>>>>>>>>>>>>>>>> the UDF project. Everyone’s welcome to drop in and share ideas! Here is the
>>>>>>>>>>>>>>>>>>>>> meeting link:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to
>>>>>>>>>>>>>>>>>>>>>> discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for
>>>>>>>>>>>>>>>>>>>>>> UDFs at this stage? We explored the possibility of simplifying the
>>>>>>>>>>>>>>>>>>>>>> specification by avoiding view replication, and potentially introducing
>>>>>>>>>>>>>>>>>>>>>> versioning support later. UDTFs, being a superset of views in some ways,
>>>>>>>>>>>>>>>>>>>>>> may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may
>>>>>>>>>>>>>>>>>>>>>> not support vararg syntax in CREATE FUNCTION,
>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs could represent such arguments as lists when supported by the
>>>>>>>>>>>>>>>>>>>>>> engine.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently
>>>>>>>>>>>>>>>>>>>>>> doesn’t support generic types (e.g., object), we can
>>>>>>>>>>>>>>>>>>>>>> only map engine-specific types to Iceberg types. As a result, generic data
>>>>>>>>>>>>>>>>>>>>>> types will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a
>>>>>>>>>>>>>>>>>>>>>> language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two primary
>>>>>>>>>>>>>>>>>>>>>> UDF use cases:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax examples
>>>>>>>>>>>>>>>>>>>>>> in both SQL and Python, and detail how each use case is represented in
>>>>>>>>>>>>>>>>>>>>>> Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the
>>>>>>>>>>>>>>>>>>>>>>> previous comments. Additionally, I've included the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>>>>> by various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and support,
>>>>>>>>>>>>>>>>>>>>>>>> I’d like to take ownership of this effort and revive the work. I'll be
>>>>>>>>>>>>>>>>>>>>>>>> revisiting the document I proposed long back and will share an updated
>>>>>>>>>>>>>>>>>>>>>>>> proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to
>>>>>>>>>>>>>>>>>>>>>>>>> be SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new
>>>>>>>>>>>>>>>>>>>>>>>>> type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the
>>>>>>>>>>>>>>>>>>>>>>>>>> spec. It leaves a way for future versions to add different representations
>>>>>>>>>>>>>>>>>>>>>>>>>> later, but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only
>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL representations, although it is certainly faviouring SQL in examples...
>>>>>>>>>>>>>>>>>>>>>>>>>>> It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal focuses on SQL-based engines, while Python-based systems often
>>>>>>>>>>>>>>>>>>>>>>>>>>>> work with data frames. Adding imperative languages like Python would make
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-created functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> statement look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> added comments. I think it would be helpful to also have a design doc that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures is limiting. It would be nice to see a discussion of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the trade-offs and a rationale for the choice. I think it would also be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very helpful to have a few representative use cases for this included in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc. That way the proposal can demonstrate that it solves those use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple function signatures per verison, but it is unclear how to encode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating functions across engines, so this doesn’t show that the metadata
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shows a SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long term opportunities in this case. Consider you register a Spark temp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view as some sort of data frame read, then it could still be resolved to a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark plan that is representable by an intermediate representation. But I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> agree this gets very complicated very soon, and just having the case (1)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benny Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports SQL representations of UDFs (similar to views as shared by the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reference links above), the complexity involved will be similar to managing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a common set of functions across engines, I don't see how that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views? I feel we can say exactly the same thing for Iceberg views, but yet
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we have Iceberg multi-dialect views implemented. Maybe it sounds like we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support is a good idea, even just a multi-dialect one like view, and that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> when we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Robert Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and portable and "non-centralized" as views are. The same performance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concerns apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's a good idea to add UDFs tracked by Iceberg catalogs. I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg primarily deals with things that are centralized, like tables of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 8:10 AM Ajantha Bhat <aj...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> addition for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

Thanks a lot for joining today's UDF sync. Here is the summary:

   1. Instead of relying on dynamic inference, the return table’s schema
   for user-defined table functions (UDTFs) should be explicitly defined.
   2. Whether a function is a UDTF should be captured as a dedicated
   attribute, rather than being inferred indirectly from the return type.
   3. The interpretation of a UDF body (whether it is treated as a partial
   SQL expression or as a full SELECT statement) should be determined by
   engines. Example: `SELECT x +1` vs. `x + 1`. Different engines have
   different takes.
   4. User-defined aggregation functions (UDAFs) are out of scope for now.
   5. Each overload should include its own current-version field. This
   avoids relying solely on the global `definition-versions` when querying the
   current version of one overload.

You can watch the recording here:https://www.youtube.com/watch?v=9t2xev8WfAw
I will update the PR(https://github.com/apache/iceberg/pull/14117) shortly.

Yufei


On Fri, Sep 19, 2025 at 9:42 AM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> Really appreciated feedback from you all over the past few months. I've
> filed the initial PR for the UDF spec:
> https://github.com/apache/iceberg/pull/14117. It captures the consensus
> we've built and addresses the write amplification concern raised in our
> last discussion.
>
> Please take a look and share your thoughts. Happy to discuss it further
> during Monday's meeting as well.
>
> Yufei
>
>
> On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks, thanks for joining today’s UDF sync.
>>
>> We covered the UDF metadata structure, captured in this doc:
>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>> .
>>
>> We also discussed a way to avoid copying every overload into the new
>> metadata JSON when creating a new version. One of ideas is to introduce a
>> global version array, this is not yet reflected in the doc, but I’ll update
>> it shortly. Other key points:
>>
>>    - The latest UDF version will typically be used in most scenarios,
>>    but engines retain the flexibility to choose which version to execute.
>>    - Keeping the version while referring to an UDF probably isn't a good
>>    idea. Users are responsible for updating downstream views if they reference
>>    older UDF versions.
>>
>> You can watch the recording here:
>> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>>
>> Yufei
>>
>>
>> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks, thanks for attending today’s UDF sync. In general, we
>>> discussed the UDF metadata structure, captured at this doc(
>>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>>> ). Here is the detailed summary:
>>>
>>>    1. Each UDF overload has its own return type. e.g., `add(int, int)`
>>>    returns `int`, while `add(long, long)`  returns `long`
>>>    2. Return type should be explicitly specified, no implicit or
>>>    statement-based return type inference should be allowed.
>>>    3. Adding explicit properties like deterministic, doc properties at
>>>    the overload level.
>>>    4. Adding property “secure” at the top level.
>>>    5. Introducing a dedicated signature definitions section to
>>>    centralize metadata (Function parameters, Return type, Parameter
>>>    descriptions). Each overload would reference a signature definition by ID.
>>>    This decoupling allows signature-related updates (like modifying parameter
>>>    descriptions) without requiring a new UDF version, similar to how updating
>>>    a table schema doesn’t create a new snapshot.
>>>    6. Whether to have versioned open properties or not. Versioned
>>>    properties can lead to unnecessary copying of a bag of properties into each
>>>    version, while it provides a clear history of properties for any future
>>>    debugging and understanding of the UDF behavior at a specific point in
>>>    time.
>>>
>>> Watch the recording here,
>>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>>
>>> Yufei
>>>
>>>
>>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi everyone, here’s the summary from our last sync on 8/11. Apologies
>>>> for the delay!
>>>>
>>>>    - One UDF entity for all overloads
>>>>       - We agreed to combine overloads with the same name into a
>>>>       single UDF entity, which shares a common metadata.json file.
>>>>       - Listing UDFs will return a list of UDF names, not a list of
>>>>       individual signatures.
>>>>       - Loading a UDF by name will return all of its overloads.
>>>>    - Versioning Strategy
>>>>       - A global version number will track changes across the entire
>>>>       UDF entity, it increments monolithically.
>>>>       - Each overload will also maintain its own version (e.g.,
>>>>       updated_at_version) to trace changes specific to that overload.
>>>>    - For simplicity, the load API will not support argument-based
>>>>    filtering in the initial release. It will always return all overloads for a
>>>>    given UDF name, overload-level loading is not supported at this stage.
>>>>
>>>> Watch the recording here,
>>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> To recap and add my thoughts, we want to support UDFs with multiple
>>>>> signatures under the same name, which can serve both overload-aware and
>>>>> overload-naive engines.
>>>>>
>>>>> Per my investigation[1], most engines support overloading by arguments
>>>>> and allow implicit conversions like numeric widening (e.g., INT →
>>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>>> change. Here is an example:
>>>>>
>>>>>    - Initially, only foo(DOUBLE) exists.
>>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>>    - Later: malicious user creates foo(BIGINT).
>>>>>    - Engine’s best-match resolution now binds the same call to the
>>>>>    new overload, changing behavior without modifying the query.
>>>>>
>>>>> To mitigate this issue, we have to choose between these two access
>>>>> control models:
>>>>>
>>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads of a
>>>>>    function name.
>>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>>    signatures.
>>>>>
>>>>> The general recommendation is to adopt *Model A.* It trades some
>>>>> precision for safety and simplicity, while eliminating the silent behavior
>>>>> change problem. More details are in this doc[1].
>>>>>
>>>>> 1.
>>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <aj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks to everyone who joined the sync.
>>>>>> Here is the meeting recording:
>>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>>
>>>>>> Summary:
>>>>>> We have discussed how to define function identifiers (should also
>>>>>> handle function overloading). Ryan suggested that we should check how Spark
>>>>>> does it. We can refer to functions using an identifier and then bind the
>>>>>> different signatures to it. So that access policies can be applied per
>>>>>> identifier. This is also linked to how we want to version the functions
>>>>>> when overloading is supported.
>>>>>>
>>>>>> I will check more about this and update the proposal doc.
>>>>>>
>>>>>> Please check/subscribe to the dev events calendar for the next
>>>>>> meeting link (Aug 11).
>>>>>>
>>>>>> - Ajantha
>>>>>>
>>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Ajantha,
>>>>>>>
>>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev Events"
>>>>>>> calendar for tomorrow 7/28 at 9AM PT. I missed the last one, but
>>>>>>> i'll be at this one.
>>>>>>>
>>>>>>> Best,
>>>>>>> Kevin Liu
>>>>>>>
>>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey everyone,
>>>>>>>>
>>>>>>>> No one joined the sync today. I came to know that Yufei is on
>>>>>>>> holiday, and Ryan and others couldn't make it, similar to the last sync. It
>>>>>>>> seems Yufei might have forgotten to transfer meeting ownership as well, as
>>>>>>>> new members needed admin approval and couldn't join automatically this
>>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>>
>>>>>>>> I've updated the function signature schema and other open points. I
>>>>>>>> believe we're very close to the final version of the spec. A meeting is
>>>>>>>> indeed necessary to finalize this, but we don't have to wait for it to
>>>>>>>> finish the review process. We had many meetings on this in the past
>>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>>
>>>>>>>> - Ajantha
>>>>>>>>
>>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I’d propose to move the field `properties` from a top level field
>>>>>>>>> to a field inside “version” along with a representation, so that properties
>>>>>>>>> are versioned. A property like “deterministic” could change along with
>>>>>>>>> representation over time. For example, we need to change “deterministic”
>>>>>>>>> from true to false in case of adding a non-deterministic SQL
>>>>>>>>> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
>>>>>>>>> be safe.
>>>>>>>>>
>>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>
>>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>>
>>>>>>>>>>    1. Each version still enforces a single signature. Although
>>>>>>>>>>    the signatures array is global to the UDF, each version references just one
>>>>>>>>>>    signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>>    2. We’ve separated the less frequently changing component
>>>>>>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>>>>>>    file size.
>>>>>>>>>>    3. Since signatures use Iceberg data types, they should
>>>>>>>>>>    remain unaffected by multi-dialect representation differences.
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>>
>>>>>>>>>>> Summary:
>>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>>
>>>>>>>>>>>    - Function overloading: Supported by few of the engines and
>>>>>>>>>>>    in the roadmaps of many engines. Iceberg will support it. We will maintain
>>>>>>>>>>>    the `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>>>>>>>>>    containing the function argument's type list). And all operations like
>>>>>>>>>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag, we need
>>>>>>>>>>>    to standardize the property name. Iceberg encryption may be orthogonal to
>>>>>>>>>>>    this discussion.
>>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>>
>>>>>>>>>>> new discussions around
>>>>>>>>>>>
>>>>>>>>>>>    - Standardizing the property names (deterministic, secure).
>>>>>>>>>>>    - About the rename function.
>>>>>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>>>>>    supported (considering function overloading) .
>>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>>
>>>>>>>>>>>    I think we are close on the spec. Please review the proposal
>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>>    .
>>>>>>>>>>>
>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>
>>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>
>>>>>>>>>>> - Ajantha
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole metadata
>>>>>>>>>>>> is encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>>>>>>>>>> discuss more on the sync today.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the extra detail. What do you think the spec would
>>>>>>>>>>>>>> require? Would it require hiding the UDF definition from users and require
>>>>>>>>>>>>>> specific pushdown cases be disabled? The use cases seem valid, but I'm
>>>>>>>>>>>>>> trying to understand the requirements this places on engines and why it
>>>>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <
>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the UDF
>>>>>>>>>>>>>>>    body and details like the list of imports, some of them aren’t applicable
>>>>>>>>>>>>>>>    to SQL UDFs.
>>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an
>>>>>>>>>>>>>>>    isolated environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For example,
>>>>>>>>>>>>>>>    secure UDFs may disable certain optimizations—such as predicate pushdown—to
>>>>>>>>>>>>>>>    avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Given these scenarios, I agree with your point that the
>>>>>>>>>>>>>>> secure flag is primarily an instruction to the engine to
>>>>>>>>>>>>>>> behave differently. While it's largely an engine-side behavior, we still
>>>>>>>>>>>>>>> need to include this flag in the UDF definition to indicate whether a UDF
>>>>>>>>>>>>>>> is secure, especially considering the perf penalty introduced by scenario
>>>>>>>>>>>>>>> #3. We should clearly recommend that users avoid marking UDFs as secure
>>>>>>>>>>>>>>> unless it's truly necessary.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a
>>>>>>>>>>>>>>>> "secure" UDF? What use case are you addressing and what specifically
>>>>>>>>>>>>>>>> changes about how the UDF is handled? If the idea is to hide the UDF
>>>>>>>>>>>>>>>> definition, do we need to include it?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think this would be a signal to a "trusted engine". When
>>>>>>>>>>>>>>>> the engine interacts with the catalog it sends authorization information
>>>>>>>>>>>>>>>> about itself in addition to the user that it is acting on behalf of. That
>>>>>>>>>>>>>>>> way the catalog knows that the secure UDF can be sent to the engine and
>>>>>>>>>>>>>>>> won't be shown to the user. The majority of this logic is on the REST
>>>>>>>>>>>>>>>> server side, and the only part that is communicated to the client is the
>>>>>>>>>>>>>>>> request not to show the UDF to the user, right? In that case should this be
>>>>>>>>>>>>>>>> a property rather than part of the definition? Even if we state that the
>>>>>>>>>>>>>>>> client "must" suppress the UDF definition, it's really just a request. Only
>>>>>>>>>>>>>>>> trusted engines can be passed the UDF definition, so a spec requirement to
>>>>>>>>>>>>>>>> suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether
>>>>>>>>>>>>>>>>> those statements run within a single transaction should be treated as an
>>>>>>>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure UDF”
>>>>>>>>>>>>>>>>> option supported by some engines[1], so the body and any sensitive details
>>>>>>>>>>>>>>>>> stay hidden from callers?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>    by different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator, like $$
>>>>>>>>>>>>>>>>>>    or '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to check
>>>>>>>>>>>>>>>>>>    whether these engines support function overloading for SQL UDFs. Postgres
>>>>>>>>>>>>>>>>>>    supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>>> ready.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    We discussed including Python support; the majority
>>>>>>>>>>>>>>>>>>>    agreed *not to* (see recording for details).
>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be
>>>>>>>>>>>>>>>>>>>    included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each catalog
>>>>>>>>>>>>>>>>>>>    resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing
>>>>>>>>>>>>>>>>>>>    table/view spec — e.g., location will remain
>>>>>>>>>>>>>>>>>>>    *required* for cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    We also discussed a bit about view interoperability
>>>>>>>>>>>>>>>>>>>    as the same things are applicable here.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>>>>    ready.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for
>>>>>>>>>>>>>>>>>>>> the UDF project. Everyone’s welcome to drop in and share ideas! Here is the
>>>>>>>>>>>>>>>>>>>> meeting link:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to
>>>>>>>>>>>>>>>>>>>>> discuss the UDF proposal. We covered several key points, though some are
>>>>>>>>>>>>>>>>>>>>> still open for further discussion:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for
>>>>>>>>>>>>>>>>>>>>> UDFs at this stage? We explored the possibility of simplifying the
>>>>>>>>>>>>>>>>>>>>> specification by avoiding view replication, and potentially introducing
>>>>>>>>>>>>>>>>>>>>> versioning support later. UDTFs, being a superset of views in some ways,
>>>>>>>>>>>>>>>>>>>>> may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may
>>>>>>>>>>>>>>>>>>>>> not support vararg syntax in CREATE FUNCTION, Iceberg
>>>>>>>>>>>>>>>>>>>>> UDFs could represent such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently
>>>>>>>>>>>>>>>>>>>>> doesn’t support generic types (e.g., object), we can
>>>>>>>>>>>>>>>>>>>>> only map engine-specific types to Iceberg types. As a result, generic data
>>>>>>>>>>>>>>>>>>>>> types will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a
>>>>>>>>>>>>>>>>>>>>> language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two primary
>>>>>>>>>>>>>>>>>>>>> UDF use cases:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax examples
>>>>>>>>>>>>>>>>>>>>> in both SQL and Python, and detail how each use case is represented in
>>>>>>>>>>>>>>>>>>>>> Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the
>>>>>>>>>>>>>>>>>>>>>> previous comments. Additionally, I've included the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>>>> by various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and support, I’d
>>>>>>>>>>>>>>>>>>>>>>> like to take ownership of this effort and revive the work. I'll be
>>>>>>>>>>>>>>>>>>>>>>> revisiting the document I proposed long back and will share an updated
>>>>>>>>>>>>>>>>>>>>>>> proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to be
>>>>>>>>>>>>>>>>>>>>>>>> SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new
>>>>>>>>>>>>>>>>>>>>>>>> type in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the
>>>>>>>>>>>>>>>>>>>>>>>>> spec. It leaves a way for future versions to add different representations
>>>>>>>>>>>>>>>>>>>>>>>>> later, but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri
>>>>>>>>>>>>>>>>>>>>>>>>> Bourlatchkov <dm...@dremio.com.invalid>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only
>>>>>>>>>>>>>>>>>>>>>>>>>> SQL representations, although it is certainly faviouring SQL in examples...
>>>>>>>>>>>>>>>>>>>>>>>>>> It would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this
>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal focuses on SQL-based engines, while Python-based systems often
>>>>>>>>>>>>>>>>>>>>>>>>>>> work with data frames. Adding imperative languages like Python would make
>>>>>>>>>>>>>>>>>>>>>>>>>>> this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>> Findeisen <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> added comments. I think it would be helpful to also have a design doc that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> covers the choices from the draft spec. For instance, the choice to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enumerate all possible function input struts rather than allowing generics
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures is limiting. It would be nice to see a discussion of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the trade-offs and a rationale for the choice. I think it would also be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> very helpful to have a few representative use cases for this included in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc. That way the proposal can demonstrate that it solves those use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple function signatures per verison, but it is unclear how to encode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating functions across engines, so this doesn’t show that the metadata
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shows a SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> didn't get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long term opportunities in this case. Consider you register a Spark temp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> view as some sort of data frame read, then it could still be resolved to a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark plan that is representable by an intermediate representation. But I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> agree this gets very complicated very soon, and just having the case (1)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benny Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports SQL representations of UDFs (similar to views as shared by the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reference links above), the complexity involved will be similar to managing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a common set of functions across engines, I don't see how that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> views? I feel we can say exactly the same thing for Iceberg views, but yet
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we have Iceberg multi-dialect views implemented. Maybe it sounds like we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support is a good idea, even just a multi-dialect one like view, and that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Robert Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> portable and "non-centralized" as views are. The same performance concerns
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should also try to tackle the idea to make views portable, which is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conceptually not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's a good idea to add UDFs tracked by Iceberg catalogs. I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg primarily deals with things that are centralized, like tables of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data. While it would be great to have a common set of functions across
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines, I don't see how that is practical when those engines are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented so differently. Plugging in code -- and especially custom
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user-supplied code -- seems inherently specialized to me and should be part
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> addition for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dremio, Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

Really appreciated feedback from you all over the past few months. I've
filed the initial PR for the UDF spec:
https://github.com/apache/iceberg/pull/14117. It captures the consensus
we've built and addresses the write amplification concern raised in our
last discussion.

Please take a look and share your thoughts. Happy to discuss it further
during Monday's meeting as well.

Yufei


On Mon, Sep 8, 2025 at 6:33 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks, thanks for joining today’s UDF sync.
>
> We covered the UDF metadata structure, captured in this doc:
> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
> .
>
> We also discussed a way to avoid copying every overload into the new
> metadata JSON when creating a new version. One of ideas is to introduce a
> global version array, this is not yet reflected in the doc, but I’ll update
> it shortly. Other key points:
>
>    - The latest UDF version will typically be used in most scenarios, but
>    engines retain the flexibility to choose which version to execute.
>    - Keeping the version while referring to an UDF probably isn't a good
>    idea. Users are responsible for updating downstream views if they reference
>    older UDF versions.
>
> You can watch the recording here:
> https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg
>
> Yufei
>
>
> On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks, thanks for attending today’s UDF sync. In general, we discussed
>> the UDF metadata structure, captured at this doc(
>> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
>> ). Here is the detailed summary:
>>
>>    1. Each UDF overload has its own return type. e.g., `add(int, int)`
>>    returns `int`, while `add(long, long)`  returns `long`
>>    2. Return type should be explicitly specified, no implicit or
>>    statement-based return type inference should be allowed.
>>    3. Adding explicit properties like deterministic, doc properties at
>>    the overload level.
>>    4. Adding property “secure” at the top level.
>>    5. Introducing a dedicated signature definitions section to
>>    centralize metadata (Function parameters, Return type, Parameter
>>    descriptions). Each overload would reference a signature definition by ID.
>>    This decoupling allows signature-related updates (like modifying parameter
>>    descriptions) without requiring a new UDF version, similar to how updating
>>    a table schema doesn’t create a new snapshot.
>>    6. Whether to have versioned open properties or not. Versioned
>>    properties can lead to unnecessary copying of a bag of properties into each
>>    version, while it provides a clear history of properties for any future
>>    debugging and understanding of the UDF behavior at a specific point in
>>    time.
>>
>> Watch the recording here,
>> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>>
>> Yufei
>>
>>
>> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi everyone, here’s the summary from our last sync on 8/11. Apologies
>>> for the delay!
>>>
>>>    - One UDF entity for all overloads
>>>       - We agreed to combine overloads with the same name into a single
>>>       UDF entity, which shares a common metadata.json file.
>>>       - Listing UDFs will return a list of UDF names, not a list of
>>>       individual signatures.
>>>       - Loading a UDF by name will return all of its overloads.
>>>    - Versioning Strategy
>>>       - A global version number will track changes across the entire
>>>       UDF entity, it increments monolithically.
>>>       - Each overload will also maintain its own version (e.g.,
>>>       updated_at_version) to trace changes specific to that overload.
>>>    - For simplicity, the load API will not support argument-based
>>>    filtering in the initial release. It will always return all overloads for a
>>>    given UDF name, overload-level loading is not supported at this stage.
>>>
>>> Watch the recording here,
>>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>>
>>> Yufei
>>>
>>>
>>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> To recap and add my thoughts, we want to support UDFs with multiple
>>>> signatures under the same name, which can serve both overload-aware and
>>>> overload-naive engines.
>>>>
>>>> Per my investigation[1], most engines support overloading by arguments
>>>> and allow implicit conversions like numeric widening (e.g., INT →
>>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>>> change. Here is an example:
>>>>
>>>>    - Initially, only foo(DOUBLE) exists.
>>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>>    - Later: malicious user creates foo(BIGINT).
>>>>    - Engine’s best-match resolution now binds the same call to the new
>>>>    overload, changing behavior without modifying the query.
>>>>
>>>> To mitigate this issue, we have to choose between these two access
>>>> control models:
>>>>
>>>>    1. Model A – Name-Level ACL: Grants apply to all overloads of a
>>>>    function name.
>>>>    2. Model B – Signature-Level ACL: Grants tied to specific
>>>>    signatures.
>>>>
>>>> The general recommendation is to adopt *Model A.* It trades some
>>>> precision for safety and simplicity, while eliminating the silent behavior
>>>> change problem. More details are in this doc[1].
>>>>
>>>> 1.
>>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <aj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks to everyone who joined the sync.
>>>>> Here is the meeting recording:
>>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>>
>>>>> Summary:
>>>>> We have discussed how to define function identifiers (should also
>>>>> handle function overloading). Ryan suggested that we should check how Spark
>>>>> does it. We can refer to functions using an identifier and then bind the
>>>>> different signatures to it. So that access policies can be applied per
>>>>> identifier. This is also linked to how we want to version the functions
>>>>> when overloading is supported.
>>>>>
>>>>> I will check more about this and update the proposal doc.
>>>>>
>>>>> Please check/subscribe to the dev events calendar for the next
>>>>> meeting link (Aug 11).
>>>>>
>>>>> - Ajantha
>>>>>
>>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Ajantha,
>>>>>>
>>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev Events"
>>>>>> calendar for tomorrow 7/28 at 9AM PT. I missed the last one, but
>>>>>> i'll be at this one.
>>>>>>
>>>>>> Best,
>>>>>> Kevin Liu
>>>>>>
>>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <aj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey everyone,
>>>>>>>
>>>>>>> No one joined the sync today. I came to know that Yufei is on
>>>>>>> holiday, and Ryan and others couldn't make it, similar to the last sync. It
>>>>>>> seems Yufei might have forgotten to transfer meeting ownership as well, as
>>>>>>> new members needed admin approval and couldn't join automatically this
>>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>>
>>>>>>> I've updated the function signature schema and other open points. I
>>>>>>> believe we're very close to the final version of the spec. A meeting is
>>>>>>> indeed necessary to finalize this, but we don't have to wait for it to
>>>>>>> finish the review process. We had many meetings on this in the past
>>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>>
>>>>>>> - Ajantha
>>>>>>>
>>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I’d propose to move the field `properties` from a top level field
>>>>>>>> to a field inside “version” along with a representation, so that properties
>>>>>>>> are versioned. A property like “deterministic” could change along with
>>>>>>>> representation over time. For example, we need to change “deterministic”
>>>>>>>> from true to false in case of adding a non-deterministic SQL
>>>>>>>> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
>>>>>>>> be safe.
>>>>>>>>
>>>>>>>> That said, it's still an open question whether we need any
>>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>
>>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>>> representation history. Here are reasons:
>>>>>>>>>
>>>>>>>>>    1. Each version still enforces a single signature. Although
>>>>>>>>>    the signatures array is global to the UDF, each version references just one
>>>>>>>>>    signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>>    2. We’ve separated the less frequently changing component
>>>>>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>>>>>    file size.
>>>>>>>>>    3. Since signatures use Iceberg data types, they should remain
>>>>>>>>>    unaffected by multi-dialect representation differences.
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>>
>>>>>>>>>> Summary:
>>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>>
>>>>>>>>>>    - Function overloading: Supported by few of the engines and
>>>>>>>>>>    in the roadmaps of many engines. Iceberg will support it. We will maintain
>>>>>>>>>>    the `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>>>>>>>>    containing the function argument's type list). And all operations like
>>>>>>>>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>>>>    - Secure UDF: If we store it as a property in a bag, we need
>>>>>>>>>>    to standardize the property name. Iceberg encryption may be orthogonal to
>>>>>>>>>>    this discussion.
>>>>>>>>>>    - UDF with multi statement and procedural bodies are
>>>>>>>>>>    supported by some engines. Iceberg will support it. Store the body as it is
>>>>>>>>>>    while creating function by the engine.
>>>>>>>>>>
>>>>>>>>>> new discussions around
>>>>>>>>>>
>>>>>>>>>>    - Standardizing the property names (deterministic, secure).
>>>>>>>>>>    - About the rename function.
>>>>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>>>>    supported (considering function overloading) .
>>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>>
>>>>>>>>>>    I think we are close on the spec. Please review the proposal
>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>>    .
>>>>>>>>>>
>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>
>>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>>>> Google Meet joining info
>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>
>>>>>>>>>> - Ajantha
>>>>>>>>>>
>>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole metadata
>>>>>>>>>>> is encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>>>>>>>>> discuss more on the sync today.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the extra detail. What do you think the spec would
>>>>>>>>>>>>> require? Would it require hiding the UDF definition from users and require
>>>>>>>>>>>>> specific pushdown cases be disabled? The use cases seem valid, but I'm
>>>>>>>>>>>>> trying to understand the requirements this places on engines and why it
>>>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the UDF
>>>>>>>>>>>>>>    body and details like the list of imports, some of them aren’t applicable
>>>>>>>>>>>>>>    to SQL UDFs.
>>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>>>>>>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For example,
>>>>>>>>>>>>>>    secure UDFs may disable certain optimizations—such as predicate pushdown—to
>>>>>>>>>>>>>>    avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Given these scenarios, I agree with your point that the
>>>>>>>>>>>>>> secure flag is primarily an instruction to the engine to
>>>>>>>>>>>>>> behave differently. While it's largely an engine-side behavior, we still
>>>>>>>>>>>>>> need to include this flag in the UDF definition to indicate whether a UDF
>>>>>>>>>>>>>> is secure, especially considering the perf penalty introduced by scenario
>>>>>>>>>>>>>> #3. We should clearly recommend that users avoid marking UDFs as secure
>>>>>>>>>>>>>> unless it's truly necessary.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a "secure"
>>>>>>>>>>>>>>> UDF? What use case are you addressing and what specifically changes about
>>>>>>>>>>>>>>> how the UDF is handled? If the idea is to hide the UDF definition, do we
>>>>>>>>>>>>>>> need to include it?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think this would be a signal to a "trusted engine". When
>>>>>>>>>>>>>>> the engine interacts with the catalog it sends authorization information
>>>>>>>>>>>>>>> about itself in addition to the user that it is acting on behalf of. That
>>>>>>>>>>>>>>> way the catalog knows that the secure UDF can be sent to the engine and
>>>>>>>>>>>>>>> won't be shown to the user. The majority of this logic is on the REST
>>>>>>>>>>>>>>> server side, and the only part that is communicated to the client is the
>>>>>>>>>>>>>>> request not to show the UDF to the user, right? In that case should this be
>>>>>>>>>>>>>>> a property rather than part of the definition? Even if we state that the
>>>>>>>>>>>>>>> client "must" suppress the UDF definition, it's really just a request. Only
>>>>>>>>>>>>>>> trusted engines can be passed the UDF definition, so a spec requirement to
>>>>>>>>>>>>>>> suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether
>>>>>>>>>>>>>>>> those statements run within a single transaction should be treated as an
>>>>>>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure UDF”
>>>>>>>>>>>>>>>> option supported by some engines[1], so the body and any sensitive details
>>>>>>>>>>>>>>>> stay hidden from callers?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>>>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>>>>>>>    - Each engine uses its own block separator, like $$ or
>>>>>>>>>>>>>>>>>    '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to check
>>>>>>>>>>>>>>>>>    whether these engines support function overloading for SQL UDFs. Postgres
>>>>>>>>>>>>>>>>>    supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>> ready.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    We discussed including Python support; the majority
>>>>>>>>>>>>>>>>>>    agreed *not to* (see recording for details).
>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be
>>>>>>>>>>>>>>>>>>    included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    Suggestions were made to document how each catalog
>>>>>>>>>>>>>>>>>>    resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing table/view
>>>>>>>>>>>>>>>>>>    spec — e.g., location will remain *required* for
>>>>>>>>>>>>>>>>>>    cross-catalog compatibility.
>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    We also discussed a bit about view interoperability
>>>>>>>>>>>>>>>>>>    as the same things are applicable here.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>>>    ready.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for
>>>>>>>>>>>>>>>>>>> the UDF project. Everyone’s welcome to drop in and share ideas! Here is the
>>>>>>>>>>>>>>>>>>> meeting link:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss
>>>>>>>>>>>>>>>>>>>> the UDF proposal. We covered several key points, though some are still open
>>>>>>>>>>>>>>>>>>>> for further discussion:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for
>>>>>>>>>>>>>>>>>>>> UDFs at this stage? We explored the possibility of simplifying the
>>>>>>>>>>>>>>>>>>>> specification by avoiding view replication, and potentially introducing
>>>>>>>>>>>>>>>>>>>> versioning support later. UDTFs, being a superset of views in some ways,
>>>>>>>>>>>>>>>>>>>> may not require versioning initially.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not
>>>>>>>>>>>>>>>>>>>> support vararg syntax in CREATE FUNCTION, Iceberg UDFs
>>>>>>>>>>>>>>>>>>>> could represent such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t
>>>>>>>>>>>>>>>>>>>> support generic types (e.g., object), we can only map
>>>>>>>>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a
>>>>>>>>>>>>>>>>>>>> language for SQL UDFs seems promising, especially given its potential to
>>>>>>>>>>>>>>>>>>>> resolve interoperability challenges. Some engines, however, require
>>>>>>>>>>>>>>>>>>>> platform version and package dependency details to execute Python code—this
>>>>>>>>>>>>>>>>>>>> should be captured in the specification.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>>> I will update the proposal document with two primary
>>>>>>>>>>>>>>>>>>>> UDF use cases:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax examples
>>>>>>>>>>>>>>>>>>>> in both SQL and Python, and detail how each use case is represented in
>>>>>>>>>>>>>>>>>>>> Iceberg metadata.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the
>>>>>>>>>>>>>>>>>>>>> previous comments. Additionally, I've included the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>>> by various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and support, I’d
>>>>>>>>>>>>>>>>>>>>>> like to take ownership of this effort and revive the work. I'll be
>>>>>>>>>>>>>>>>>>>>>> revisiting the document I proposed long back and will share an updated
>>>>>>>>>>>>>>>>>>>>>> proposal by next week.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to be
>>>>>>>>>>>>>>>>>>>>>>> SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new type
>>>>>>>>>>>>>>>>>>>>>>> in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the
>>>>>>>>>>>>>>>>>>>>>>>> spec. It leaves a way for future versions to add different representations
>>>>>>>>>>>>>>>>>>>>>>>> later, but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this
>>>>>>>>>>>>>>>>>>>>>>>>>> proposal focuses on SQL-based engines, while Python-based systems often
>>>>>>>>>>>>>>>>>>>>>>>>>> work with data frames. Adding imperative languages like Python would make
>>>>>>>>>>>>>>>>>>>>>>>>>> this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to
>>>>>>>>>>>>>>>>>>>>>>>>>>> share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement
>>>>>>>>>>>>>>>>>>>>>>>>>>>> look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate
>>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures is limiting. It would be nice to see a discussion of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the trade-offs and a rationale for the choice. I think it would also be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> very helpful to have a few representative use cases for this included in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the doc. That way the proposal can demonstrate that it solves those use
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cases with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>> multiple function signatures per verison, but it is unclear how to encode
>>>>>>>>>>>>>>>>>>>>>>>>>>>> them because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating functions across engines, so this doesn’t show that the metadata
>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function
>>>>>>>>>>>>>>>>>>>>>>>>>>>> shows a SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more
>>>>>>>>>>>>>>>>>>>>>>>>>>>> review comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>> get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some
>>>>>>>>>>>>>>>>>>>>>>>>>>>> long term opportunities in this case. Consider you register a Spark temp
>>>>>>>>>>>>>>>>>>>>>>>>>>>> view as some sort of data frame read, then it could still be resolved to a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark plan that is representable by an intermediate representation. But I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> agree this gets very complicated very soon, and just having the case (1)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa Eldin Moustafa <wa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about
>>>>>>>>>>>>>>>>>>>>>>>>>>>> what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially
>>>>>>>>>>>>>>>>>>>>>>>>>>>> supports SQL representations of UDFs (similar to views as shared by the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> reference links above), the complexity involved will be similar to managing
>>>>>>>>>>>>>>>>>>>>>>>>>>>> views.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a common set of functions across engines, I don't see how that is practical
>>>>>>>>>>>>>>>>>>>>>>>>>>>> when those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> views? I feel we can say exactly the same thing for Iceberg views, but yet
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we have Iceberg multi-dialect views implemented. Maybe it sounds like we
>>>>>>>>>>>>>>>>>>>>>>>>>>>> are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>> support is a good idea, even just a multi-dialect one like view, and that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> can allow engines to for example parse a view SQL, and when a function
>>>>>>>>>>>>>>>>>>>>>>>>>>>> referenced cannot be resolved, try to seek for a multi-dialect UDF
>>>>>>>>>>>>>>>>>>>>>>>>>>>> definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Robert Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> portable and "non-centralized" as views are. The same performance concerns
>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common
>>>>>>>>>>>>>>>>>>>>>>>>>>>> base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>>>>>> also try to tackle the idea to make views portable, which is conceptually
>>>>>>>>>>>>>>>>>>>>>>>>>>>> not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a good idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>>> addition for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate
>>>>>>>>>>>>>>>>>>>>>>>>>>>> similarly to views in that they are associated with tables, but they can
>>>>>>>>>>>>>>>>>>>>>>>>>>>> accept arguments and produce return values, or even function as inline
>>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can
>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun
>>>>>>>>>>>>>>>>>>>>>>>>>>>> drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks, thanks for joining today’s UDF sync.

We covered the UDF metadata structure, captured in this doc:
https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
.

We also discussed a way to avoid copying every overload into the new
metadata JSON when creating a new version. One of ideas is to introduce a
global version array, this is not yet reflected in the doc, but I’ll update
it shortly. Other key points:

   - The latest UDF version will typically be used in most scenarios, but
   engines retain the flexibility to choose which version to execute.
   - Keeping the version while referring to an UDF probably isn't a good
   idea. Users are responsible for updating downstream views if they reference
   older UDF versions.

You can watch the recording here:
https://www.youtube.com/watch?v=6ResT-ODelI&ab_channel=ApacheIceberg

Yufei


On Mon, Aug 25, 2025 at 6:36 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks, thanks for attending today’s UDF sync. In general, we discussed
> the UDF metadata structure, captured at this doc(
> https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
> ). Here is the detailed summary:
>
>    1. Each UDF overload has its own return type. e.g., `add(int, int)`
>    returns `int`, while `add(long, long)`  returns `long`
>    2. Return type should be explicitly specified, no implicit or
>    statement-based return type inference should be allowed.
>    3. Adding explicit properties like deterministic, doc properties at
>    the overload level.
>    4. Adding property “secure” at the top level.
>    5. Introducing a dedicated signature definitions section to centralize
>    metadata (Function parameters, Return type, Parameter descriptions). Each
>    overload would reference a signature definition by ID. This decoupling
>    allows signature-related updates (like modifying parameter descriptions)
>    without requiring a new UDF version, similar to how updating a table schema
>    doesn’t create a new snapshot.
>    6. Whether to have versioned open properties or not. Versioned
>    properties can lead to unnecessary copying of a bag of properties into each
>    version, while it provides a clear history of properties for any future
>    debugging and understanding of the UDF behavior at a specific point in
>    time.
>
> Watch the recording here,
> https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL
>
> Yufei
>
>
> On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi everyone, here’s the summary from our last sync on 8/11. Apologies for
>> the delay!
>>
>>    - One UDF entity for all overloads
>>       - We agreed to combine overloads with the same name into a single
>>       UDF entity, which shares a common metadata.json file.
>>       - Listing UDFs will return a list of UDF names, not a list of
>>       individual signatures.
>>       - Loading a UDF by name will return all of its overloads.
>>    - Versioning Strategy
>>       - A global version number will track changes across the entire UDF
>>       entity, it increments monolithically.
>>       - Each overload will also maintain its own version (e.g.,
>>       updated_at_version) to trace changes specific to that overload.
>>    - For simplicity, the load API will not support argument-based
>>    filtering in the initial release. It will always return all overloads for a
>>    given UDF name, overload-level loading is not supported at this stage.
>>
>> Watch the recording here,
>> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>>
>> Yufei
>>
>>
>> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> To recap and add my thoughts, we want to support UDFs with multiple
>>> signatures under the same name, which can serve both overload-aware and
>>> overload-naive engines.
>>>
>>> Per my investigation[1], most engines support overloading by arguments
>>> and allow implicit conversions like numeric widening (e.g., INT →
>>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>>> change. Here is an example:
>>>
>>>    - Initially, only foo(DOUBLE) exists.
>>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>>    - Later: malicious user creates foo(BIGINT).
>>>    - Engine’s best-match resolution now binds the same call to the new
>>>    overload, changing behavior without modifying the query.
>>>
>>> To mitigate this issue, we have to choose between these two access
>>> control models:
>>>
>>>    1. Model A – Name-Level ACL: Grants apply to all overloads of a
>>>    function name.
>>>    2. Model B – Signature-Level ACL: Grants tied to specific signatures.
>>>
>>> The general recommendation is to adopt *Model A.* It trades some
>>> precision for safety and simplicity, while eliminating the silent behavior
>>> change problem. More details are in this doc[1].
>>>
>>> 1.
>>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>>
>>> Yufei
>>>
>>>
>>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <aj...@gmail.com>
>>> wrote:
>>>
>>>> Thanks to everyone who joined the sync.
>>>> Here is the meeting recording:
>>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>>
>>>> Summary:
>>>> We have discussed how to define function identifiers (should also
>>>> handle function overloading). Ryan suggested that we should check how Spark
>>>> does it. We can refer to functions using an identifier and then bind the
>>>> different signatures to it. So that access policies can be applied per
>>>> identifier. This is also linked to how we want to version the functions
>>>> when overloading is supported.
>>>>
>>>> I will check more about this and update the proposal doc.
>>>>
>>>> Please check/subscribe to the dev events calendar for the next
>>>> meeting link (Aug 11).
>>>>
>>>> - Ajantha
>>>>
>>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Ajantha,
>>>>>
>>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev Events"
>>>>> calendar for tomorrow 7/28 at 9AM PT. I missed the last one, but i'll
>>>>> be at this one.
>>>>>
>>>>> Best,
>>>>> Kevin Liu
>>>>>
>>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <aj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hey everyone,
>>>>>>
>>>>>> No one joined the sync today. I came to know that Yufei is on
>>>>>> holiday, and Ryan and others couldn't make it, similar to the last sync. It
>>>>>> seems Yufei might have forgotten to transfer meeting ownership as well, as
>>>>>> new members needed admin approval and couldn't join automatically this
>>>>>> week. Also, I can understand it is summer holiday season for many.
>>>>>>
>>>>>> I've updated the function signature schema and other open points. I
>>>>>> believe we're very close to the final version of the spec. A meeting is
>>>>>> indeed necessary to finalize this, but we don't have to wait for it to
>>>>>> finish the review process. We had many meetings on this in the past
>>>>>> already. So, please review the document at your earliest convenience. If we
>>>>>> agree on the spec by next week, I can raise a PR.
>>>>>>
>>>>>> - Ajantha
>>>>>>
>>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>>>
>>>>>>> I’d propose to move the field `properties` from a top level field to
>>>>>>> a field inside “version” along with a representation, so that properties
>>>>>>> are versioned. A property like “deterministic” could change along with
>>>>>>> representation over time. For example, we need to change “deterministic”
>>>>>>> from true to false in case of adding a non-deterministic SQL
>>>>>>> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
>>>>>>> be safe.
>>>>>>>
>>>>>>> That said, it's still an open question whether we need any
>>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>
>>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>>> representation history. Here are reasons:
>>>>>>>>
>>>>>>>>    1. Each version still enforces a single signature. Although the
>>>>>>>>    signatures array is global to the UDF, each version references just one
>>>>>>>>    signature ID. Rollbacks to historical versions remain safe.
>>>>>>>>    2. We’ve separated the less frequently changing component
>>>>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>>>>    file size.
>>>>>>>>    3. Since signatures use Iceberg data types, they should remain
>>>>>>>>    unaffected by multi-dialect representation differences.
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>> Here is the meeting recording:
>>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>>
>>>>>>>>> Summary:
>>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>>
>>>>>>>>>    - Function overloading: Supported by few of the engines and in
>>>>>>>>>    the roadmaps of many engines. Iceberg will support it. We will maintain the
>>>>>>>>>    `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>>>>>>>    containing the function argument's type list). And all operations like
>>>>>>>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>>>    - Secure UDF: If we store it as a property in a bag, we need
>>>>>>>>>    to standardize the property name. Iceberg encryption may be orthogonal to
>>>>>>>>>    this discussion.
>>>>>>>>>    - UDF with multi statement and procedural bodies are supported
>>>>>>>>>    by some engines. Iceberg will support it. Store the body as it is while
>>>>>>>>>    creating function by the engine.
>>>>>>>>>
>>>>>>>>> new discussions around
>>>>>>>>>
>>>>>>>>>    - Standardizing the property names (deterministic, secure).
>>>>>>>>>    - About the rename function.
>>>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>>>    supported (considering function overloading) .
>>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>>
>>>>>>>>>    I think we are close on the spec. Please review the proposal
>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>>    .
>>>>>>>>>
>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>
>>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>>> Google Meet joining info
>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>
>>>>>>>>> - Ajantha
>>>>>>>>>
>>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Can it be handled by Iceberg encryption? If the whole metadata is
>>>>>>>>>> encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>>>>>>>> discuss more on the sync today.
>>>>>>>>>>
>>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, hiding the definition and disabling pushdown are
>>>>>>>>>>> required.We will need a named key(e.g., secure) somewhere, no matter if it
>>>>>>>>>>> is a top level property or a key as a part of the UDF properties. So that
>>>>>>>>>>> both UDF creator and consumer can recognize it.
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the extra detail. What do you think the spec would
>>>>>>>>>>>> require? Would it require hiding the UDF definition from users and require
>>>>>>>>>>>> specific pushdown cases be disabled? The use cases seem valid, but I'm
>>>>>>>>>>>> trying to understand the requirements this places on engines and why it
>>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    1.
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the UDF
>>>>>>>>>>>>>    body and details like the list of imports, some of them aren’t applicable
>>>>>>>>>>>>>    to SQL UDFs.
>>>>>>>>>>>>>    2.
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>>>>>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>>    3.
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For example,
>>>>>>>>>>>>>    secure UDFs may disable certain optimizations—such as predicate pushdown—to
>>>>>>>>>>>>>    avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Given these scenarios, I agree with your point that the secure
>>>>>>>>>>>>> flag is primarily an instruction to the engine to behave differently. While
>>>>>>>>>>>>> it's largely an engine-side behavior, we still need to include this flag in
>>>>>>>>>>>>> the UDF definition to indicate whether a UDF is secure, especially
>>>>>>>>>>>>> considering the perf penalty introduced by scenario #3. We should clearly
>>>>>>>>>>>>> recommend that users avoid marking UDFs as secure unless it's truly
>>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei, could you make the argument for supporting a "secure"
>>>>>>>>>>>>>> UDF? What use case are you addressing and what specifically changes about
>>>>>>>>>>>>>> how the UDF is handled? If the idea is to hide the UDF definition, do we
>>>>>>>>>>>>>> need to include it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think this would be a signal to a "trusted engine". When
>>>>>>>>>>>>>> the engine interacts with the catalog it sends authorization information
>>>>>>>>>>>>>> about itself in addition to the user that it is acting on behalf of. That
>>>>>>>>>>>>>> way the catalog knows that the secure UDF can be sent to the engine and
>>>>>>>>>>>>>> won't be shown to the user. The majority of this logic is on the REST
>>>>>>>>>>>>>> server side, and the only part that is communicated to the client is the
>>>>>>>>>>>>>> request not to show the UDF to the user, right? In that case should this be
>>>>>>>>>>>>>> a property rather than part of the definition? Even if we state that the
>>>>>>>>>>>>>> client "must" suppress the UDF definition, it's really just a request. Only
>>>>>>>>>>>>>> trusted engines can be passed the UDF definition, so a spec requirement to
>>>>>>>>>>>>>> suppress the definition isn't very meaningful.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <
>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether
>>>>>>>>>>>>>>> those statements run within a single transaction should be treated as an
>>>>>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> One more thing: should we also introduce a “secure UDF”
>>>>>>>>>>>>>>> option supported by some engines[1], so the body and any sensitive details
>>>>>>>>>>>>>>> stay hidden from callers?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>>>>>>    - Each engine uses its own block separator, like $$ or
>>>>>>>>>>>>>>>>    '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to check
>>>>>>>>>>>>>>>>    whether these engines support function overloading for SQL UDFs. Postgres
>>>>>>>>>>>>>>>>    supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    We discussed including Python support; the majority
>>>>>>>>>>>>>>>>>    agreed *not to* (see recording for details).
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be
>>>>>>>>>>>>>>>>>    included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Suggestions were made to document how each catalog
>>>>>>>>>>>>>>>>>    resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing table/view
>>>>>>>>>>>>>>>>>    spec — e.g., location will remain *required* for
>>>>>>>>>>>>>>>>>    cross-catalog compatibility.
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    We also discussed a bit about view interoperability as
>>>>>>>>>>>>>>>>>    the same things are applicable here.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>>    ready.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the
>>>>>>>>>>>>>>>>>> UDF project. Everyone’s welcome to drop in and share ideas! Here is the
>>>>>>>>>>>>>>>>>> meeting link:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss
>>>>>>>>>>>>>>>>>>> the UDF proposal. We covered several key points, though some are still open
>>>>>>>>>>>>>>>>>>> for further discussion:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for
>>>>>>>>>>>>>>>>>>> UDFs at this stage? We explored the possibility of simplifying the
>>>>>>>>>>>>>>>>>>> specification by avoiding view replication, and potentially introducing
>>>>>>>>>>>>>>>>>>> versioning support later. UDTFs, being a superset of views in some ways,
>>>>>>>>>>>>>>>>>>> may not require versioning initially.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not
>>>>>>>>>>>>>>>>>>> support vararg syntax in CREATE FUNCTION, Iceberg UDFs
>>>>>>>>>>>>>>>>>>> could represent such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t
>>>>>>>>>>>>>>>>>>> support generic types (e.g., object), we can only map
>>>>>>>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a language
>>>>>>>>>>>>>>>>>>> for SQL UDFs seems promising, especially given its potential to resolve
>>>>>>>>>>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>>>>>>>>>>> be captured in the specification.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>>> I will update the proposal document with two primary UDF
>>>>>>>>>>>>>>>>>>> use cases:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The update will include corresponding syntax examples in
>>>>>>>>>>>>>>>>>>> both SQL and Python, and detail how each use case is represented in Iceberg
>>>>>>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the
>>>>>>>>>>>>>>>>>>>> previous comments. Additionally, I've included the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>>> by various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Given the strong community interest and support, I’d
>>>>>>>>>>>>>>>>>>>>> like to take ownership of this effort and revive the work. I'll be
>>>>>>>>>>>>>>>>>>>>> revisiting the document I proposed long back and will share an updated
>>>>>>>>>>>>>>>>>>>>> proposal by next week.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to be
>>>>>>>>>>>>>>>>>>>>>> SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new type
>>>>>>>>>>>>>>>>>>>>>> in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the
>>>>>>>>>>>>>>>>>>>>>>> spec. It leaves a way for future versions to add different representations
>>>>>>>>>>>>>>>>>>>>>>> later, but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this
>>>>>>>>>>>>>>>>>>>>>>>>> proposal focuses on SQL-based engines, while Python-based systems often
>>>>>>>>>>>>>>>>>>>>>>>>> work with data frames. Adding imperative languages like Python would make
>>>>>>>>>>>>>>>>>>>>>>>>> this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread
>>>>>>>>>>>>>>>>>>>>>>>>>> [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to
>>>>>>>>>>>>>>>>>>>>>>>>>> share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>> The Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement
>>>>>>>>>>>>>>>>>>>>>>>>>>> look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple
>>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures per verison, but it is unclear how to encode them
>>>>>>>>>>>>>>>>>>>>>>>>>>> because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows
>>>>>>>>>>>>>>>>>>>>>>>>>>> a SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on
>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a
>>>>>>>>>>>>>>>>>>>>>>>>>>> look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't
>>>>>>>>>>>>>>>>>>>>>>>>>>> get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far
>>>>>>>>>>>>>>>>>>>>>>>>>>> (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the
>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>> <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some
>>>>>>>>>>>>>>>>>>>>>>>>>>> long term opportunities in this case. Consider you register a Spark temp
>>>>>>>>>>>>>>>>>>>>>>>>>>> view as some sort of data frame read, then it could still be resolved to a
>>>>>>>>>>>>>>>>>>>>>>>>>>> Spark plan that is representable by an intermediate representation. But I
>>>>>>>>>>>>>>>>>>>>>>>>>>> agree this gets very complicated very soon, and just having the case (1)
>>>>>>>>>>>>>>>>>>>>>>>>>>> covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a
>>>>>>>>>>>>>>>>>>>>>>>>>>> tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about
>>>>>>>>>>>>>>>>>>>>>>>>>>> what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by
>>>>>>>>>>>>>>>>>>>>>>>>>>> the user whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in
>>>>>>>>>>>>>>>>>>>>>>>>>>> imperative function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports
>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL representations of UDFs (similar to views as shared by the reference
>>>>>>>>>>>>>>>>>>>>>>>>>>> links above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for
>>>>>>>>>>>>>>>>>>>>>>>>>>> your input.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the
>>>>>>>>>>>>>>>>>>>>>>>>>>> draft spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jack Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have
>>>>>>>>>>>>>>>>>>>>>>>>>>> a common set of functions across engines, I don't see how that is practical
>>>>>>>>>>>>>>>>>>>>>>>>>>> when those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the
>>>>>>>>>>>>>>>>>>>>>>>>>>> views? I feel we can say exactly the same thing for Iceberg views, but yet
>>>>>>>>>>>>>>>>>>>>>>>>>>> we have Iceberg multi-dialect views implemented. Maybe it sounds like we
>>>>>>>>>>>>>>>>>>>>>>>>>>> are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support
>>>>>>>>>>>>>>>>>>>>>>>>>>> is a good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when
>>>>>>>>>>>>>>>>>>>>>>>>>>> we have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>> Robert Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and
>>>>>>>>>>>>>>>>>>>>>>>>>>> portable and "non-centralized" as views are. The same performance concerns
>>>>>>>>>>>>>>>>>>>>>>>>>>> apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common
>>>>>>>>>>>>>>>>>>>>>>>>>>> base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>>>>> also try to tackle the idea to make views portable, which is conceptually
>>>>>>>>>>>>>>>>>>>>>>>>>>> not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a
>>>>>>>>>>>>>>>>>>>>>>>>>>> negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's
>>>>>>>>>>>>>>>>>>>>>>>>>>> a good idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM
>>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge
>>>>>>>>>>>>>>>>>>>>>>>>>>> the community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec
>>>>>>>>>>>>>>>>>>>>>>>>>>> addition for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly
>>>>>>>>>>>>>>>>>>>>>>>>>>> to views in that they are associated with tables, but they can accept
>>>>>>>>>>>>>>>>>>>>>>>>>>> arguments and produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can
>>>>>>>>>>>>>>>>>>>>>>>>>>> enable
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the
>>>>>>>>>>>>>>>>>>>>>>>>>>> engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this
>>>>>>>>>>>>>>>>>>>>>>>>>>> feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun
>>>>>>>>>>>>>>>>>>>>>>>>>>> drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on
>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks, thanks for attending today’s UDF sync. In general, we discussed
the UDF metadata structure, captured at this doc(
https://docs.google.com/document/d/1khPKL6zvWjYc5Is8HeVau6sff8FD-jNc2eLKXgit3X8/edit?usp=sharing
). Here is the detailed summary:

   1. Each UDF overload has its own return type. e.g., `add(int, int)`
   returns `int`, while `add(long, long)`  returns `long`
   2. Return type should be explicitly specified, no implicit or
   statement-based return type inference should be allowed.
   3. Adding explicit properties like deterministic, doc properties at the
   overload level.
   4. Adding property “secure” at the top level.
   5. Introducing a dedicated signature definitions section to centralize
   metadata (Function parameters, Return type, Parameter descriptions). Each
   overload would reference a signature definition by ID. This decoupling
   allows signature-related updates (like modifying parameter descriptions)
   without requiring a new UDF version, similar to how updating a table schema
   doesn’t create a new snapshot.
   6. Whether to have versioned open properties or not. Versioned
   properties can lead to unnecessary copying of a bag of properties into each
   version, while it provides a clear history of properties for any future
   debugging and understanding of the UDF behavior at a specific point in
   time.

Watch the recording here,
https://www.youtube.com/watch?v=p7CvuGZKLSo&list=PLkifVhhWtccwzc3oRWjy5XiYJl0R6kdQL

Yufei


On Thu, Aug 21, 2025 at 4:18 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi everyone, here’s the summary from our last sync on 8/11. Apologies for
> the delay!
>
>    - One UDF entity for all overloads
>       - We agreed to combine overloads with the same name into a single
>       UDF entity, which shares a common metadata.json file.
>       - Listing UDFs will return a list of UDF names, not a list of
>       individual signatures.
>       - Loading a UDF by name will return all of its overloads.
>    - Versioning Strategy
>       - A global version number will track changes across the entire UDF
>       entity, it increments monolithically.
>       - Each overload will also maintain its own version (e.g.,
>       updated_at_version) to trace changes specific to that overload.
>    - For simplicity, the load API will not support argument-based
>    filtering in the initial release. It will always return all overloads for a
>    given UDF name, overload-level loading is not supported at this stage.
>
> Watch the recording here,
> https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view
>
> Yufei
>
>
> On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> To recap and add my thoughts, we want to support UDFs with multiple
>> signatures under the same name, which can serve both overload-aware and
>> overload-naive engines.
>>
>> Per my investigation[1], most engines support overloading by arguments
>> and allow implicit conversions like numeric widening (e.g., INT →
>> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
>> change. Here is an example:
>>
>>    - Initially, only foo(DOUBLE) exists.
>>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>>    - Later: malicious user creates foo(BIGINT).
>>    - Engine’s best-match resolution now binds the same call to the new
>>    overload, changing behavior without modifying the query.
>>
>> To mitigate this issue, we have to choose between these two access
>> control models:
>>
>>    1. Model A – Name-Level ACL: Grants apply to all overloads of a
>>    function name.
>>    2. Model B – Signature-Level ACL: Grants tied to specific signatures.
>>
>> The general recommendation is to adopt *Model A.* It trades some
>> precision for safety and simplicity, while eliminating the silent behavior
>> change problem. More details are in this doc[1].
>>
>> 1.
>> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>>
>> Yufei
>>
>>
>> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <aj...@gmail.com>
>> wrote:
>>
>>> Thanks to everyone who joined the sync.
>>> Here is the meeting recording:
>>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>>
>>> Summary:
>>> We have discussed how to define function identifiers (should also handle
>>> function overloading). Ryan suggested that we should check how Spark does
>>> it. We can refer to functions using an identifier and then bind the
>>> different signatures to it. So that access policies can be applied per
>>> identifier. This is also linked to how we want to version the functions
>>> when overloading is supported.
>>>
>>> I will check more about this and update the proposal doc.
>>>
>>> Please check/subscribe to the dev events calendar for the next
>>> meeting link (Aug 11).
>>>
>>> - Ajantha
>>>
>>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org>
>>> wrote:
>>>
>>>> Hi Ajantha,
>>>>
>>>> I see that the UDF Sync is scheduled in the "Iceberg Dev Events"
>>>> calendar for tomorrow 7/28 at 9AM PT. I missed the last one, but i'll
>>>> be at this one.
>>>>
>>>> Best,
>>>> Kevin Liu
>>>>
>>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <aj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> No one joined the sync today. I came to know that Yufei is on holiday,
>>>>> and Ryan and others couldn't make it, similar to the last sync. It seems
>>>>> Yufei might have forgotten to transfer meeting ownership as well, as new
>>>>> members needed admin approval and couldn't join automatically this week.
>>>>> Also, I can understand it is summer holiday season for many.
>>>>>
>>>>> I've updated the function signature schema and other open points. I
>>>>> believe we're very close to the final version of the spec. A meeting is
>>>>> indeed necessary to finalize this, but we don't have to wait for it to
>>>>> finish the review process. We had many meetings on this in the past
>>>>> already. So, please review the document at your earliest convenience. If we
>>>>> agree on the spec by next week, I can raise a PR.
>>>>>
>>>>> - Ajantha
>>>>>
>>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> I’d propose to move the field `properties` from a top level field to
>>>>>> a field inside “version” along with a representation, so that properties
>>>>>> are versioned. A property like “deterministic” could change along with
>>>>>> representation over time. For example, we need to change “deterministic”
>>>>>> from true to false in case of adding a non-deterministic SQL
>>>>>> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
>>>>>> be safe.
>>>>>>
>>>>>> That said, it's still an open question whether we need any
>>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>
>>>>>>> I’d prefer to keep the signature list separate from the
>>>>>>> representation history. Here are reasons:
>>>>>>>
>>>>>>>    1. Each version still enforces a single signature. Although the
>>>>>>>    signatures array is global to the UDF, each version references just one
>>>>>>>    signature ID. Rollbacks to historical versions remain safe.
>>>>>>>    2. We’ve separated the less frequently changing component
>>>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>>>    file size.
>>>>>>>    3. Since signatures use Iceberg data types, they should remain
>>>>>>>    unaffected by multi-dialect representation differences.
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>> Here is the meeting recording:
>>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>>
>>>>>>>> Summary:
>>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>>> Appendix C* in the proposal doc)
>>>>>>>>
>>>>>>>>    - Function overloading: Supported by few of the engines and in
>>>>>>>>    the roadmaps of many engines. Iceberg will support it. We will maintain the
>>>>>>>>    `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>>>>>>    containing the function argument's type list). And all operations like
>>>>>>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>>    - Secure UDF: If we store it as a property in a bag, we need to
>>>>>>>>    standardize the property name. Iceberg encryption may be orthogonal to this
>>>>>>>>    discussion.
>>>>>>>>    - UDF with multi statement and procedural bodies are supported
>>>>>>>>    by some engines. Iceberg will support it. Store the body as it is while
>>>>>>>>    creating function by the engine.
>>>>>>>>
>>>>>>>> new discussions around
>>>>>>>>
>>>>>>>>    - Standardizing the property names (deterministic, secure).
>>>>>>>>    - About the rename function.
>>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>>    supported (considering function overloading) .
>>>>>>>>    - Signature should be associated with representation?
>>>>>>>>
>>>>>>>>    I think we are close on the spec. Please review the proposal
>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>>    .
>>>>>>>>
>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>
>>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>> Google Meet joining info
>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>
>>>>>>>> - Ajantha
>>>>>>>>
>>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <aj...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Can it be handled by Iceberg encryption? If the whole metadata is
>>>>>>>>> encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>>>>>>> discuss more on the sync today.
>>>>>>>>>
>>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, hiding the definition and disabling pushdown are required.We
>>>>>>>>>> will need a named key(e.g., secure) somewhere, no matter if it is a top
>>>>>>>>>> level property or a key as a part of the UDF properties. So that both UDF
>>>>>>>>>> creator and consumer can recognize it.
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the extra detail. What do you think the spec would
>>>>>>>>>>> require? Would it require hiding the UDF definition from users and require
>>>>>>>>>>> specific pushdown cases be disabled? The use cases seem valid, but I'm
>>>>>>>>>>> trying to understand the requirements this places on engines and why it
>>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>
>>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>>
>>>>>>>>>>>>    1.
>>>>>>>>>>>>
>>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the UDF
>>>>>>>>>>>>    body and details like the list of imports, some of them aren’t applicable
>>>>>>>>>>>>    to SQL UDFs.
>>>>>>>>>>>>    2.
>>>>>>>>>>>>
>>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>>>>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>>    3.
>>>>>>>>>>>>
>>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For example,
>>>>>>>>>>>>    secure UDFs may disable certain optimizations—such as predicate pushdown—to
>>>>>>>>>>>>    avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>>
>>>>>>>>>>>> Given these scenarios, I agree with your point that the secure
>>>>>>>>>>>> flag is primarily an instruction to the engine to behave differently. While
>>>>>>>>>>>> it's largely an engine-side behavior, we still need to include this flag in
>>>>>>>>>>>> the UDF definition to indicate whether a UDF is secure, especially
>>>>>>>>>>>> considering the perf penalty introduced by scenario #3. We should clearly
>>>>>>>>>>>> recommend that users avoid marking UDFs as secure unless it's truly
>>>>>>>>>>>> necessary.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei, could you make the argument for supporting a "secure"
>>>>>>>>>>>>> UDF? What use case are you addressing and what specifically changes about
>>>>>>>>>>>>> how the UDF is handled? If the idea is to hide the UDF definition, do we
>>>>>>>>>>>>> need to include it?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think this would be a signal to a "trusted engine". When the
>>>>>>>>>>>>> engine interacts with the catalog it sends authorization information about
>>>>>>>>>>>>> itself in addition to the user that it is acting on behalf of. That way the
>>>>>>>>>>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>>>>>>>>>>> shown to the user. The majority of this logic is on the REST server side,
>>>>>>>>>>>>> and the only part that is communicated to the client is the request not to
>>>>>>>>>>>>> show the UDF to the user, right? In that case should this be a property
>>>>>>>>>>>>> rather than part of the definition? Even if we state that the client "must"
>>>>>>>>>>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>>>>>>>>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>>>>>>>>>>> the definition isn't very meaningful.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>>>>>>>>>>> statements run within a single transaction should be treated as an
>>>>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> One more thing: should we also introduce a “secure UDF”
>>>>>>>>>>>>>> option supported by some engines[1], so the body and any sensitive details
>>>>>>>>>>>>>> stay hidden from callers?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>>>>>    - Each engine uses its own block separator, like $$ or
>>>>>>>>>>>>>>>    '' or none. Action item was to check whether engines support
>>>>>>>>>>>>>>>    multi-statement (transactional) UDF bodies.
>>>>>>>>>>>>>>>    - Discussed about function overloading. Need to check
>>>>>>>>>>>>>>>    whether these engines support function overloading for SQL UDFs. Postgres
>>>>>>>>>>>>>>>    supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    We discussed including Python support; the majority
>>>>>>>>>>>>>>>>    agreed *not to* (see recording for details).
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be
>>>>>>>>>>>>>>>>    included to support change tracking and similar use cases.
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Suggestions were made to document how each catalog
>>>>>>>>>>>>>>>>    resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    We agreed not to deviate from the existing table/view
>>>>>>>>>>>>>>>>    spec — e.g., location will remain *required* for
>>>>>>>>>>>>>>>>    cross-catalog compatibility.
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    We also discussed a bit about view interoperability as
>>>>>>>>>>>>>>>>    the same things are applicable here.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>>    Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>>    ready.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the
>>>>>>>>>>>>>>>>> UDF project. Everyone’s welcome to drop in and share ideas! Here is the
>>>>>>>>>>>>>>>>> meeting link:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss
>>>>>>>>>>>>>>>>>> the UDF proposal. We covered several key points, though some are still open
>>>>>>>>>>>>>>>>>> for further discussion:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for
>>>>>>>>>>>>>>>>>> UDFs at this stage? We explored the possibility of simplifying the
>>>>>>>>>>>>>>>>>> specification by avoiding view replication, and potentially introducing
>>>>>>>>>>>>>>>>>> versioning support later. UDTFs, being a superset of views in some ways,
>>>>>>>>>>>>>>>>>> may not require versioning initially.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not
>>>>>>>>>>>>>>>>>> support vararg syntax in CREATE FUNCTION, Iceberg UDFs
>>>>>>>>>>>>>>>>>> could represent such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t
>>>>>>>>>>>>>>>>>> support generic types (e.g., object), we can only map
>>>>>>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a language
>>>>>>>>>>>>>>>>>> for SQL UDFs seems promising, especially given its potential to resolve
>>>>>>>>>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>>>>>>>>>> be captured in the specification.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>>> I will update the proposal document with two primary UDF
>>>>>>>>>>>>>>>>>> use cases:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The update will include corresponding syntax examples in
>>>>>>>>>>>>>>>>>> both SQL and Python, and detail how each use case is represented in Iceberg
>>>>>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the
>>>>>>>>>>>>>>>>>>> previous comments. Additionally, I've included the SQL UDF syntax supported
>>>>>>>>>>>>>>>>>>> by various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Given the strong community interest and support, I’d
>>>>>>>>>>>>>>>>>>>> like to take ownership of this effort and revive the work. I'll be
>>>>>>>>>>>>>>>>>>>> revisiting the document I proposed long back and will share an updated
>>>>>>>>>>>>>>>>>>>> proposal by next week.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to be
>>>>>>>>>>>>>>>>>>>>> SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new type
>>>>>>>>>>>>>>>>>>>>> in the "Representations" section).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the
>>>>>>>>>>>>>>>>>>>>>> spec. It leaves a way for future versions to add different representations
>>>>>>>>>>>>>>>>>>>>>> later, but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this
>>>>>>>>>>>>>>>>>>>>>>>> proposal focuses on SQL-based engines, while Python-based systems often
>>>>>>>>>>>>>>>>>>>>>>>> work with data frames. Adding imperative languages like Python would make
>>>>>>>>>>>>>>>>>>>>>>>> this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread
>>>>>>>>>>>>>>>>>>>>>>>>> [1] i read
>>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to
>>>>>>>>>>>>>>>>>>>>>>>>> share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The
>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement
>>>>>>>>>>>>>>>>>>>>>>>>>> look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple
>>>>>>>>>>>>>>>>>>>>>>>>>> function signatures per verison, but it is unclear how to encode them
>>>>>>>>>>>>>>>>>>>>>>>>>> because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows
>>>>>>>>>>>>>>>>>>>>>>>>>> a SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on
>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a
>>>>>>>>>>>>>>>>>>>>>>>>>> look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't
>>>>>>>>>>>>>>>>>>>>>>>>>> get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far
>>>>>>>>>>>>>>>>>>>>>>>>>> (from Benny).
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the
>>>>>>>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some
>>>>>>>>>>>>>>>>>>>>>>>>>> long term opportunities in this case. Consider you register a Spark temp
>>>>>>>>>>>>>>>>>>>>>>>>>> view as some sort of data frame read, then it could still be resolved to a
>>>>>>>>>>>>>>>>>>>>>>>>>> Spark plan that is representable by an intermediate representation. But I
>>>>>>>>>>>>>>>>>>>>>>>>>> agree this gets very complicated very soon, and just having the case (1)
>>>>>>>>>>>>>>>>>>>>>>>>>> covered would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny
>>>>>>>>>>>>>>>>>>>>>>>>>> Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a
>>>>>>>>>>>>>>>>>>>>>>>>>> tabular SQL UDF can be used to build a parameterized view.  So, there's
>>>>>>>>>>>>>>>>>>>>>>>>>> definitely a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about
>>>>>>>>>>>>>>>>>>>>>>>>>> what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the
>>>>>>>>>>>>>>>>>>>>>>>>>> user whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM
>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports
>>>>>>>>>>>>>>>>>>>>>>>>>> SQL representations of UDFs (similar to views as shared by the reference
>>>>>>>>>>>>>>>>>>>>>>>>>> links above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for
>>>>>>>>>>>>>>>>>>>>>>>>>> your input.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft
>>>>>>>>>>>>>>>>>>>>>>>>>> spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack
>>>>>>>>>>>>>>>>>>>>>>>>>> Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a
>>>>>>>>>>>>>>>>>>>>>>>>>> common set of functions across engines, I don't see how that is practical
>>>>>>>>>>>>>>>>>>>>>>>>>> when those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the
>>>>>>>>>>>>>>>>>>>>>>>>>> views? I feel we can say exactly the same thing for Iceberg views, but yet
>>>>>>>>>>>>>>>>>>>>>>>>>> we have Iceberg multi-dialect views implemented. Maybe it sounds like we
>>>>>>>>>>>>>>>>>>>>>>>>>> are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support
>>>>>>>>>>>>>>>>>>>>>>>>>> is a good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we
>>>>>>>>>>>>>>>>>>>>>>>>>> have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM
>>>>>>>>>>>>>>>>>>>>>>>>>> Robert Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and
>>>>>>>>>>>>>>>>>>>>>>>>>> portable and "non-centralized" as views are. The same performance concerns
>>>>>>>>>>>>>>>>>>>>>>>>>> apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common
>>>>>>>>>>>>>>>>>>>>>>>>>> base upon which engines can build, so the argument that UDFs aren't
>>>>>>>>>>>>>>>>>>>>>>>>>> practical, because engines are different, is probably only a temporary
>>>>>>>>>>>>>>>>>>>>>>>>>> concern.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>>>> also try to tackle the idea to make views portable, which is conceptually
>>>>>>>>>>>>>>>>>>>>>>>>>> not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a
>>>>>>>>>>>>>>>>>>>>>>>>>> negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a
>>>>>>>>>>>>>>>>>>>>>>>>>> good idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM
>>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec
>>>>>>>>>>>>>>>>>>>>>>>>>> addition for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly
>>>>>>>>>>>>>>>>>>>>>>>>>> to views in that they are associated with tables, but they can accept
>>>>>>>>>>>>>>>>>>>>>>>>>> arguments and produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio,
>>>>>>>>>>>>>>>>>>>>>>>>>> Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can
>>>>>>>>>>>>>>>>>>>>>>>>>> enable
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the
>>>>>>>>>>>>>>>>>>>>>>>>>> engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this
>>>>>>>>>>>>>>>>>>>>>>>>>> feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun
>>>>>>>>>>>>>>>>>>>>>>>>>> drafting a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi everyone, here’s the summary from our last sync on 8/11. Apologies for
the delay!

   - One UDF entity for all overloads
      - We agreed to combine overloads with the same name into a single UDF
      entity, which shares a common metadata.json file.
      - Listing UDFs will return a list of UDF names, not a list of
      individual signatures.
      - Loading a UDF by name will return all of its overloads.
   - Versioning Strategy
      - A global version number will track changes across the entire UDF
      entity, it increments monolithically.
      - Each overload will also maintain its own version (e.g.,
      updated_at_version) to trace changes specific to that overload.
   - For simplicity, the load API will not support argument-based filtering
   in the initial release. It will always return all overloads for a given UDF
   name, overload-level loading is not supported at this stage.

Watch the recording here,
https://drive.google.com/file/d/10G2HjUH2DaKSjGufEOjMu0bBuNd7sCzO/view

Yufei


On Fri, Aug 8, 2025 at 3:11 PM Yufei Gu <fl...@gmail.com> wrote:

> To recap and add my thoughts, we want to support UDFs with multiple
> signatures under the same name, which can serve both overload-aware and
> overload-naive engines.
>
> Per my investigation[1], most engines support overloading by arguments and
> allow implicit conversions like numeric widening (e.g., INT →
> BIGINT/FLOAT). The resolution approach causes issues like silent behavior
> change. Here is an example:
>
>    - Initially, only foo(DOUBLE) exists.
>    - foo(42::INT) widens INT → DOUBLE and runs expected code.
>    - Later: malicious user creates foo(BIGINT).
>    - Engine’s best-match resolution now binds the same call to the new
>    overload, changing behavior without modifying the query.
>
> To mitigate this issue, we have to choose between these two access control
> models:
>
>    1. Model A – Name-Level ACL: Grants apply to all overloads of a
>    function name.
>    2. Model B – Signature-Level ACL: Grants tied to specific signatures.
>
> The general recommendation is to adopt *Model A.* It trades some
> precision for safety and simplicity, while eliminating the silent behavior
> change problem. More details are in this doc[1].
>
> 1.
> https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0
>
> Yufei
>
>
> On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
>> Thanks to everyone who joined the sync.
>> Here is the meeting recording:
>> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>>
>> Summary:
>> We have discussed how to define function identifiers (should also handle
>> function overloading). Ryan suggested that we should check how Spark does
>> it. We can refer to functions using an identifier and then bind the
>> different signatures to it. So that access policies can be applied per
>> identifier. This is also linked to how we want to version the functions
>> when overloading is supported.
>>
>> I will check more about this and update the proposal doc.
>>
>> Please check/subscribe to the dev events calendar for the next
>> meeting link (Aug 11).
>>
>> - Ajantha
>>
>> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org> wrote:
>>
>>> Hi Ajantha,
>>>
>>> I see that the UDF Sync is scheduled in the "Iceberg Dev Events"
>>> calendar for tomorrow 7/28 at 9AM PT. I missed the last one, but i'll
>>> be at this one.
>>>
>>> Best,
>>> Kevin Liu
>>>
>>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <aj...@gmail.com>
>>> wrote:
>>>
>>>> Hey everyone,
>>>>
>>>> No one joined the sync today. I came to know that Yufei is on holiday,
>>>> and Ryan and others couldn't make it, similar to the last sync. It seems
>>>> Yufei might have forgotten to transfer meeting ownership as well, as new
>>>> members needed admin approval and couldn't join automatically this week.
>>>> Also, I can understand it is summer holiday season for many.
>>>>
>>>> I've updated the function signature schema and other open points. I
>>>> believe we're very close to the final version of the spec. A meeting is
>>>> indeed necessary to finalize this, but we don't have to wait for it to
>>>> finish the review process. We had many meetings on this in the past
>>>> already. So, please review the document at your earliest convenience. If we
>>>> agree on the spec by next week, I can raise a PR.
>>>>
>>>> - Ajantha
>>>>
>>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> I’d propose to move the field `properties` from a top level field to a
>>>>> field inside “version” along with a representation, so that properties are
>>>>> versioned. A property like “deterministic” could change along with
>>>>> representation over time. For example, we need to change “deterministic”
>>>>> from true to false in case of adding a non-deterministic SQL
>>>>> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
>>>>> be safe.
>>>>>
>>>>> That said, it's still an open question whether we need any
>>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Thanks for the summary, Ajantha!
>>>>>>
>>>>>> I’d prefer to keep the signature list separate from the
>>>>>> representation history. Here are reasons:
>>>>>>
>>>>>>    1. Each version still enforces a single signature. Although the
>>>>>>    signatures array is global to the UDF, each version references just one
>>>>>>    signature ID. Rollbacks to historical versions remain safe.
>>>>>>    2. We’ve separated the less frequently changing component
>>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>>    file size.
>>>>>>    3. Since signatures use Iceberg data types, they should remain
>>>>>>    unaffected by multi-dialect representation differences.
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <aj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks to everyone who joined the sync.
>>>>>>> Here is the meeting recording:
>>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>>
>>>>>>> Summary:
>>>>>>> We have discussed the action items from the last sync (*see
>>>>>>> Appendix C* in the proposal doc)
>>>>>>>
>>>>>>>    - Function overloading: Supported by few of the engines and in
>>>>>>>    the roadmaps of many engines. Iceberg will support it. We will maintain the
>>>>>>>    `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>>>>>    containing the function argument's type list). And all operations like
>>>>>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>>    - Secure UDF: If we store it as a property in a bag, we need to
>>>>>>>    standardize the property name. Iceberg encryption may be orthogonal to this
>>>>>>>    discussion.
>>>>>>>    - UDF with multi statement and procedural bodies are supported
>>>>>>>    by some engines. Iceberg will support it. Store the body as it is while
>>>>>>>    creating function by the engine.
>>>>>>>
>>>>>>> new discussions around
>>>>>>>
>>>>>>>    - Standardizing the property names (deterministic, secure).
>>>>>>>    - About the rename function.
>>>>>>>    - Replace function. To check upto what level replace is
>>>>>>>    supported (considering function overloading) .
>>>>>>>    - Signature should be associated with representation?
>>>>>>>
>>>>>>>    I think we are close on the spec. Please review the proposal
>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>>    .
>>>>>>>
>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>
>>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>> Google Meet joining info
>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>
>>>>>>> - Ajantha
>>>>>>>
>>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Can it be handled by Iceberg encryption? If the whole metadata is
>>>>>>>> encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>>>>>> discuss more on the sync today.
>>>>>>>>
>>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, hiding the definition and disabling pushdown are required.We
>>>>>>>>> will need a named key(e.g., secure) somewhere, no matter if it is a top
>>>>>>>>> level property or a key as a part of the UDF properties. So that both UDF
>>>>>>>>> creator and consumer can recognize it.
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the extra detail. What do you think the spec would
>>>>>>>>>> require? Would it require hiding the UDF definition from users and require
>>>>>>>>>> specific pushdown cases be disabled? The use cases seem valid, but I'm
>>>>>>>>>> trying to understand the requirements this places on engines and why it
>>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>>
>>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>
>>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>>
>>>>>>>>>>>    1.
>>>>>>>>>>>
>>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the UDF
>>>>>>>>>>>    body and details like the list of imports, some of them aren’t applicable
>>>>>>>>>>>    to SQL UDFs.
>>>>>>>>>>>    2.
>>>>>>>>>>>
>>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>>>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>>    3.
>>>>>>>>>>>
>>>>>>>>>>>    Preventing Data Leakage at Execution Time: For example,
>>>>>>>>>>>    secure UDFs may disable certain optimizations—such as predicate pushdown—to
>>>>>>>>>>>    avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>>
>>>>>>>>>>> Given these scenarios, I agree with your point that the secure
>>>>>>>>>>> flag is primarily an instruction to the engine to behave differently. While
>>>>>>>>>>> it's largely an engine-side behavior, we still need to include this flag in
>>>>>>>>>>> the UDF definition to indicate whether a UDF is secure, especially
>>>>>>>>>>> considering the perf penalty introduced by scenario #3. We should clearly
>>>>>>>>>>> recommend that users avoid marking UDFs as secure unless it's truly
>>>>>>>>>>> necessary.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Yufei, could you make the argument for supporting a "secure"
>>>>>>>>>>>> UDF? What use case are you addressing and what specifically changes about
>>>>>>>>>>>> how the UDF is handled? If the idea is to hide the UDF definition, do we
>>>>>>>>>>>> need to include it?
>>>>>>>>>>>>
>>>>>>>>>>>> I think this would be a signal to a "trusted engine". When the
>>>>>>>>>>>> engine interacts with the catalog it sends authorization information about
>>>>>>>>>>>> itself in addition to the user that it is acting on behalf of. That way the
>>>>>>>>>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>>>>>>>>>> shown to the user. The majority of this logic is on the REST server side,
>>>>>>>>>>>> and the only part that is communicated to the client is the request not to
>>>>>>>>>>>> show the UDF to the user, right? In that case should this be a property
>>>>>>>>>>>> rather than part of the definition? Even if we state that the client "must"
>>>>>>>>>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>>>>>>>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>>>>>>>>>> the definition isn't very meaningful.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>>>>>>>>>> statements run within a single transaction should be treated as an
>>>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> One more thing: should we also introduce a “secure UDF” option
>>>>>>>>>>>>> supported by some engines[1], so the body and any sensitive details stay
>>>>>>>>>>>>> hidden from callers?
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>>>>    - Each engine uses its own block separator, like $$ or ''
>>>>>>>>>>>>>>    or none. Action item was to check whether engines support multi-statement
>>>>>>>>>>>>>>    (transactional) UDF bodies.
>>>>>>>>>>>>>>    - Discussed about function overloading. Need to check
>>>>>>>>>>>>>>    whether these engines support function overloading for SQL UDFs. Postgres
>>>>>>>>>>>>>>    supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    We discussed including Python support; the majority
>>>>>>>>>>>>>>>    agreed *not to* (see recording for details).
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    No strong opposition to versioning — it will be included
>>>>>>>>>>>>>>>    to support change tracking and similar use cases.
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Suggestions were made to document how each catalog
>>>>>>>>>>>>>>>    resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    We agreed not to deviate from the existing table/view
>>>>>>>>>>>>>>>    spec — e.g., location will remain *required* for
>>>>>>>>>>>>>>>    cross-catalog compatibility.
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    We also discussed a bit about view interoperability as
>>>>>>>>>>>>>>>    the same things are applicable here.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>>    Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>>    ready.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the
>>>>>>>>>>>>>>>> UDF project. Everyone’s welcome to drop in and share ideas! Here is the
>>>>>>>>>>>>>>>> meeting link:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss
>>>>>>>>>>>>>>>>> the UDF proposal. We covered several key points, though some are still open
>>>>>>>>>>>>>>>>> for further discussion:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs
>>>>>>>>>>>>>>>>> at this stage? We explored the possibility of simplifying the specification
>>>>>>>>>>>>>>>>> by avoiding view replication, and potentially introducing versioning
>>>>>>>>>>>>>>>>> support later. UDTFs, being a superset of views in some ways, may not
>>>>>>>>>>>>>>>>> require versioning initially.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not
>>>>>>>>>>>>>>>>> support vararg syntax in CREATE FUNCTION, Iceberg UDFs
>>>>>>>>>>>>>>>>> could represent such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t
>>>>>>>>>>>>>>>>> support generic types (e.g., object), we can only map
>>>>>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a language
>>>>>>>>>>>>>>>>> for SQL UDFs seems promising, especially given its potential to resolve
>>>>>>>>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>>>>>>>>> be captured in the specification.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>>> I will update the proposal document with two primary UDF
>>>>>>>>>>>>>>>>> use cases:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The update will include corresponding syntax examples in
>>>>>>>>>>>>>>>>> both SQL and Python, and detail how each use case is represented in Iceberg
>>>>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>>>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>>>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better
>>>>>>>>>>>>>>>>>>> understand the new requirement of using UDFs for fine-grained access
>>>>>>>>>>>>>>>>>>> control policies. This expands the use cases beyond just versioned and
>>>>>>>>>>>>>>>>>>> interoperable UDFs. Additionally, I learnt that many vendors are interested
>>>>>>>>>>>>>>>>>>> in this feature.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Given the strong community interest and support, I’d
>>>>>>>>>>>>>>>>>>> like to take ownership of this effort and revive the work. I'll be
>>>>>>>>>>>>>>>>>>> revisiting the document I proposed long back and will share an updated
>>>>>>>>>>>>>>>>>>> proposal by next week.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to be
>>>>>>>>>>>>>>>>>>>> SQL. It merely does not specify (in this revision) how other
>>>>>>>>>>>>>>>>>>>> representations are to be written.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new type in
>>>>>>>>>>>>>>>>>>>> the "Representations" section).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec.
>>>>>>>>>>>>>>>>>>>>> It leaves a way for future versions to add different representations later,
>>>>>>>>>>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this
>>>>>>>>>>>>>>>>>>>>>>> proposal focuses on SQL-based engines, while Python-based systems often
>>>>>>>>>>>>>>>>>>>>>>> work with data frames. Adding imperative languages like Python would make
>>>>>>>>>>>>>>>>>>>>>>> this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1]
>>>>>>>>>>>>>>>>>>>>>>>> i read
>>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to
>>>>>>>>>>>>>>>>>>>>>>>> share among different engines."
>>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The
>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement
>>>>>>>>>>>>>>>>>>>>>>>>> look like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple
>>>>>>>>>>>>>>>>>>>>>>>>> function signatures per verison, but it is unclear how to encode them
>>>>>>>>>>>>>>>>>>>>>>>>> because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a
>>>>>>>>>>>>>>>>>>>>>>>>> SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a
>>>>>>>>>>>>>>>>>>>>>>>>> look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat
>>>>>>>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't
>>>>>>>>>>>>>>>>>>>>>>>>> get any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat
>>>>>>>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from
>>>>>>>>>>>>>>>>>>>>>>>>> Benny).
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat
>>>>>>>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the
>>>>>>>>>>>>>>>>>>>>>>>>> proposal.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long
>>>>>>>>>>>>>>>>>>>>>>>>> term opportunities in this case. Consider you register a Spark temp view as
>>>>>>>>>>>>>>>>>>>>>>>>> some sort of data frame read, then it could still be resolved to a Spark
>>>>>>>>>>>>>>>>>>>>>>>>> plan that is representable by an intermediate representation. But I agree
>>>>>>>>>>>>>>>>>>>>>>>>> this gets very complicated very soon, and just having the case (1) covered
>>>>>>>>>>>>>>>>>>>>>>>>> would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny
>>>>>>>>>>>>>>>>>>>>>>>>> Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular
>>>>>>>>>>>>>>>>>>>>>>>>> SQL UDF can be used to build a parameterized view.  So, there's definitely
>>>>>>>>>>>>>>>>>>>>>>>>> a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about
>>>>>>>>>>>>>>>>>>>>>>>>> what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the
>>>>>>>>>>>>>>>>>>>>>>>>> user whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM
>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports
>>>>>>>>>>>>>>>>>>>>>>>>> SQL representations of UDFs (similar to views as shared by the reference
>>>>>>>>>>>>>>>>>>>>>>>>> links above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for
>>>>>>>>>>>>>>>>>>>>>>>>> your input.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft
>>>>>>>>>>>>>>>>>>>>>>>>> spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack
>>>>>>>>>>>>>>>>>>>>>>>>> Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a
>>>>>>>>>>>>>>>>>>>>>>>>> common set of functions across engines, I don't see how that is practical
>>>>>>>>>>>>>>>>>>>>>>>>> when those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the
>>>>>>>>>>>>>>>>>>>>>>>>> views? I feel we can say exactly the same thing for Iceberg views, but yet
>>>>>>>>>>>>>>>>>>>>>>>>> we have Iceberg multi-dialect views implemented. Maybe it sounds like we
>>>>>>>>>>>>>>>>>>>>>>>>> are trying to draw a line between SQL vs other programming language as
>>>>>>>>>>>>>>>>>>>>>>>>> "code"? but I think SQL is just another type of code, and we are already
>>>>>>>>>>>>>>>>>>>>>>>>> talking about compiling all these different code dialects to an
>>>>>>>>>>>>>>>>>>>>>>>>> intermediate representation (using projects like Coral, Substrait), which
>>>>>>>>>>>>>>>>>>>>>>>>> will be stored as another type of representation of Iceberg view. I think
>>>>>>>>>>>>>>>>>>>>>>>>> the same functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support
>>>>>>>>>>>>>>>>>>>>>>>>> is a good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we
>>>>>>>>>>>>>>>>>>>>>>>>> have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM
>>>>>>>>>>>>>>>>>>>>>>>>> Robert Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and
>>>>>>>>>>>>>>>>>>>>>>>>> portable and "non-centralized" as views are. The same performance concerns
>>>>>>>>>>>>>>>>>>>>>>>>> apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base
>>>>>>>>>>>>>>>>>>>>>>>>> upon which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>>> also try to tackle the idea to make views portable, which is conceptually
>>>>>>>>>>>>>>>>>>>>>>>>> not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a
>>>>>>>>>>>>>>>>>>>>>>>>> negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a
>>>>>>>>>>>>>>>>>>>>>>>>> good idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM
>>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec
>>>>>>>>>>>>>>>>>>>>>>>>> addition for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly
>>>>>>>>>>>>>>>>>>>>>>>>> to views in that they are associated with tables, but they can accept
>>>>>>>>>>>>>>>>>>>>>>>>> arguments and produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio,
>>>>>>>>>>>>>>>>>>>>>>>>> Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can
>>>>>>>>>>>>>>>>>>>>>>>>> enable
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the
>>>>>>>>>>>>>>>>>>>>>>>>> engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this
>>>>>>>>>>>>>>>>>>>>>>>>> feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting
>>>>>>>>>>>>>>>>>>>>>>>>> a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

To recap and add my thoughts, we want to support UDFs with multiple
signatures under the same name, which can serve both overload-aware and
overload-naive engines.

Per my investigation[1], most engines support overloading by arguments and
allow implicit conversions like numeric widening (e.g., INT →
BIGINT/FLOAT). The resolution approach causes issues like silent behavior
change. Here is an example:

   - Initially, only foo(DOUBLE) exists.
   - foo(42::INT) widens INT → DOUBLE and runs expected code.
   - Later: malicious user creates foo(BIGINT).
   - Engine’s best-match resolution now binds the same call to the new
   overload, changing behavior without modifying the query.

To mitigate this issue, we have to choose between these two access control
models:

   1. Model A – Name-Level ACL: Grants apply to all overloads of a function
   name.
   2. Model B – Signature-Level ACL: Grants tied to specific signatures.

The general recommendation is to adopt *Model A.* It trades some precision
for safety and simplicity, while eliminating the silent behavior change
problem. More details are in this doc[1].

1.
https://docs.google.com/document/d/1E8mR-vInbQ8LDa5Lv3f22i6f8sceHojnEzxEJ6s6cvc/edit?tab=t.0

Yufei


On Tue, Jul 29, 2025 at 1:07 AM Ajantha Bhat <aj...@gmail.com> wrote:

> Thanks to everyone who joined the sync.
> Here is the meeting recording:
> https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view
>
> Summary:
> We have discussed how to define function identifiers (should also handle
> function overloading). Ryan suggested that we should check how Spark does
> it. We can refer to functions using an identifier and then bind the
> different signatures to it. So that access policies can be applied per
> identifier. This is also linked to how we want to version the functions
> when overloading is supported.
>
> I will check more about this and update the proposal doc.
>
> Please check/subscribe to the dev events calendar for the next
> meeting link (Aug 11).
>
> - Ajantha
>
> On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org> wrote:
>
>> Hi Ajantha,
>>
>> I see that the UDF Sync is scheduled in the "Iceberg Dev Events"
>> calendar for tomorrow 7/28 at 9AM PT. I missed the last one, but i'll be
>> at this one.
>>
>> Best,
>> Kevin Liu
>>
>> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <aj...@gmail.com>
>> wrote:
>>
>>> Hey everyone,
>>>
>>> No one joined the sync today. I came to know that Yufei is on holiday,
>>> and Ryan and others couldn't make it, similar to the last sync. It seems
>>> Yufei might have forgotten to transfer meeting ownership as well, as new
>>> members needed admin approval and couldn't join automatically this week.
>>> Also, I can understand it is summer holiday season for many.
>>>
>>> I've updated the function signature schema and other open points. I
>>> believe we're very close to the final version of the spec. A meeting is
>>> indeed necessary to finalize this, but we don't have to wait for it to
>>> finish the review process. We had many meetings on this in the past
>>> already. So, please review the document at your earliest convenience. If we
>>> agree on the spec by next week, I can raise a PR.
>>>
>>> - Ajantha
>>>
>>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> I’d propose to move the field `properties` from a top level field to a
>>>> field inside “version” along with a representation, so that properties are
>>>> versioned. A property like “deterministic” could change along with
>>>> representation over time. For example, we need to change “deterministic”
>>>> from true to false in case of adding a non-deterministic SQL
>>>> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
>>>> be safe.
>>>>
>>>> That said, it's still an open question whether we need any
>>>> non-versioned properties. We can introduce them later if a use case arises.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Thanks for the summary, Ajantha!
>>>>>
>>>>> I’d prefer to keep the signature list separate from the representation
>>>>> history. Here are reasons:
>>>>>
>>>>>    1. Each version still enforces a single signature. Although the
>>>>>    signatures array is global to the UDF, each version references just one
>>>>>    signature ID. Rollbacks to historical versions remain safe.
>>>>>    2. We’ve separated the less frequently changing component
>>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>>    file size.
>>>>>    3. Since signatures use Iceberg data types, they should remain
>>>>>    unaffected by multi-dialect representation differences.
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <aj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks to everyone who joined the sync.
>>>>>> Here is the meeting recording:
>>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>>
>>>>>> Summary:
>>>>>> We have discussed the action items from the last sync (*see Appendix
>>>>>> C* in the proposal doc)
>>>>>>
>>>>>>    - Function overloading: Supported by few of the engines and in
>>>>>>    the roadmaps of many engines. Iceberg will support it. We will maintain the
>>>>>>    `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>>>>    containing the function argument's type list). And all operations like
>>>>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>>    - Secure UDF: If we store it as a property in a bag, we need to
>>>>>>    standardize the property name. Iceberg encryption may be orthogonal to this
>>>>>>    discussion.
>>>>>>    - UDF with multi statement and procedural bodies are supported by
>>>>>>    some engines. Iceberg will support it. Store the body as it is while
>>>>>>    creating function by the engine.
>>>>>>
>>>>>> new discussions around
>>>>>>
>>>>>>    - Standardizing the property names (deterministic, secure).
>>>>>>    - About the rename function.
>>>>>>    - Replace function. To check upto what level replace is supported
>>>>>>    (considering function overloading) .
>>>>>>    - Signature should be associated with representation?
>>>>>>
>>>>>>    I think we are close on the spec. Please review the proposal
>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>>    .
>>>>>>
>>>>>> Details for next Iceberg UDF sync:
>>>>>>
>>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>> Google Meet joining info
>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>
>>>>>> - Ajantha
>>>>>>
>>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <aj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Can it be handled by Iceberg encryption? If the whole metadata is
>>>>>>> encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>>>>> discuss more on the sync today.
>>>>>>>
>>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, hiding the definition and disabling pushdown are required.We
>>>>>>>> will need a named key(e.g., secure) somewhere, no matter if it is a top
>>>>>>>> level property or a key as a part of the UDF properties. So that both UDF
>>>>>>>> creator and consumer can recognize it.
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the extra detail. What do you think the spec would
>>>>>>>>> require? Would it require hiding the UDF definition from users and require
>>>>>>>>> specific pushdown cases be disabled? The use cases seem valid, but I'm
>>>>>>>>> trying to understand the requirements this places on engines and why it
>>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>>
>>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Ryan,
>>>>>>>>>>
>>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>>
>>>>>>>>>>    1.
>>>>>>>>>>
>>>>>>>>>>    Hiding UDF Definitions: This includes concealing the UDF body
>>>>>>>>>>    and details like the list of imports, some of them aren’t applicable to SQL
>>>>>>>>>>    UDFs.
>>>>>>>>>>    2.
>>>>>>>>>>
>>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>>    3.
>>>>>>>>>>
>>>>>>>>>>    Preventing Data Leakage at Execution Time: For example,
>>>>>>>>>>    secure UDFs may disable certain optimizations—such as predicate pushdown—to
>>>>>>>>>>    avoid exposing sensitive data indirectly. [1]
>>>>>>>>>>
>>>>>>>>>> Given these scenarios, I agree with your point that the secure
>>>>>>>>>> flag is primarily an instruction to the engine to behave differently. While
>>>>>>>>>> it's largely an engine-side behavior, we still need to include this flag in
>>>>>>>>>> the UDF definition to indicate whether a UDF is secure, especially
>>>>>>>>>> considering the perf penalty introduced by scenario #3. We should clearly
>>>>>>>>>> recommend that users avoid marking UDFs as secure unless it's truly
>>>>>>>>>> necessary.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yufei, could you make the argument for supporting a "secure"
>>>>>>>>>>> UDF? What use case are you addressing and what specifically changes about
>>>>>>>>>>> how the UDF is handled? If the idea is to hide the UDF definition, do we
>>>>>>>>>>> need to include it?
>>>>>>>>>>>
>>>>>>>>>>> I think this would be a signal to a "trusted engine". When the
>>>>>>>>>>> engine interacts with the catalog it sends authorization information about
>>>>>>>>>>> itself in addition to the user that it is acting on behalf of. That way the
>>>>>>>>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>>>>>>>>> shown to the user. The majority of this logic is on the REST server side,
>>>>>>>>>>> and the only part that is communicated to the client is the request not to
>>>>>>>>>>> show the UDF to the user, right? In that case should this be a property
>>>>>>>>>>> rather than part of the definition? Even if we state that the client "must"
>>>>>>>>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>>>>>>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>>>>>>>>> the definition isn't very meaningful.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>>
>>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>>>>>>>>> statements run within a single transaction should be treated as an
>>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>>
>>>>>>>>>>>> One more thing: should we also introduce a “secure UDF” option
>>>>>>>>>>>> supported by some engines[1], so the body and any sensitive details stay
>>>>>>>>>>>> hidden from callers?
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>>>    - Each engine uses its own block separator, like $$ or ''
>>>>>>>>>>>>>    or none. Action item was to check whether engines support multi-statement
>>>>>>>>>>>>>    (transactional) UDF bodies.
>>>>>>>>>>>>>    - Discussed about function overloading. Need to check
>>>>>>>>>>>>>    whether these engines support function overloading for SQL UDFs. Postgres
>>>>>>>>>>>>>    supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    We discussed including Python support; the majority
>>>>>>>>>>>>>>    agreed *not to* (see recording for details).
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    No strong opposition to versioning — it will be included
>>>>>>>>>>>>>>    to support change tracking and similar use cases.
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Suggestions were made to document how each catalog
>>>>>>>>>>>>>>    resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    We agreed not to deviate from the existing table/view
>>>>>>>>>>>>>>    spec — e.g., location will remain *required* for
>>>>>>>>>>>>>>    cross-catalog compatibility.
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    We also discussed a bit about view interoperability as
>>>>>>>>>>>>>>    the same things are applicable here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>>    Final spec will be put to review and vote once it is
>>>>>>>>>>>>>>    ready.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <
>>>>>>>>>>>>>> flyrain000@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the
>>>>>>>>>>>>>>> UDF project. Everyone’s welcome to drop in and share ideas! Here is the
>>>>>>>>>>>>>>> meeting link:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the
>>>>>>>>>>>>>>>> UDF proposal. We covered several key points, though some are still open for
>>>>>>>>>>>>>>>> further discussion:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs
>>>>>>>>>>>>>>>> at this stage? We explored the possibility of simplifying the specification
>>>>>>>>>>>>>>>> by avoiding view replication, and potentially introducing versioning
>>>>>>>>>>>>>>>> support later. UDTFs, being a superset of views in some ways, may not
>>>>>>>>>>>>>>>> require versioning initially.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not
>>>>>>>>>>>>>>>> support vararg syntax in CREATE FUNCTION, Iceberg UDFs
>>>>>>>>>>>>>>>> could represent such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t
>>>>>>>>>>>>>>>> support generic types (e.g., object), we can only map
>>>>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a language
>>>>>>>>>>>>>>>> for SQL UDFs seems promising, especially given its potential to resolve
>>>>>>>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>>>>>>>> be captured in the specification.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>>> I will update the proposal document with two primary UDF
>>>>>>>>>>>>>>>> use cases:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The update will include corresponding syntax examples in
>>>>>>>>>>>>>>>> both SQL and Python, and detail how each use case is represented in Iceberg
>>>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better understand
>>>>>>>>>>>>>>>>>> the new requirement of using UDFs for fine-grained access control policies.
>>>>>>>>>>>>>>>>>> This expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>>>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Given the strong community interest and support, I’d like
>>>>>>>>>>>>>>>>>> to take ownership of this effort and revive the work. I'll be revisiting
>>>>>>>>>>>>>>>>>> the document I proposed long back and will share an updated proposal by
>>>>>>>>>>>>>>>>>> next week.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to be SQL.
>>>>>>>>>>>>>>>>>>> It merely does not specify (in this revision) how other representations are
>>>>>>>>>>>>>>>>>>> to be written.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new type in
>>>>>>>>>>>>>>>>>>> the "Representations" section).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec.
>>>>>>>>>>>>>>>>>>>> It leaves a way for future versions to add different representations later,
>>>>>>>>>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this
>>>>>>>>>>>>>>>>>>>>>> proposal focuses on SQL-based engines, while Python-based systems often
>>>>>>>>>>>>>>>>>>>>>> work with data frames. Adding imperative languages like Python would make
>>>>>>>>>>>>>>>>>>>>>> this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1]
>>>>>>>>>>>>>>>>>>>>>>> i read
>>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to
>>>>>>>>>>>>>>>>>>>>>>> share among different engines."
>>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully
>>>>>>>>>>>>>>>>>>>>>>> understand yet how the proposed design addresses shareability between the
>>>>>>>>>>>>>>>>>>>>>>> engines though.
>>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The
>>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created
>>>>>>>>>>>>>>>>>>>>>>>> functions sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look
>>>>>>>>>>>>>>>>>>>>>>>> like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple
>>>>>>>>>>>>>>>>>>>>>>>> function signatures per verison, but it is unclear how to encode them
>>>>>>>>>>>>>>>>>>>>>>>> because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a
>>>>>>>>>>>>>>>>>>>>>>>> SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a
>>>>>>>>>>>>>>>>>>>>>>>> look at the proposal
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an
>>>>>>>>>>>>>>>>>>>>>>>> interesting direction, but there might be some details that need to be fine
>>>>>>>>>>>>>>>>>>>>>>>> tuned.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get
>>>>>>>>>>>>>>>>>>>>>>>> any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from
>>>>>>>>>>>>>>>>>>>>>>>> Benny).
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long
>>>>>>>>>>>>>>>>>>>>>>>> term opportunities in this case. Consider you register a Spark temp view as
>>>>>>>>>>>>>>>>>>>>>>>> some sort of data frame read, then it could still be resolved to a Spark
>>>>>>>>>>>>>>>>>>>>>>>> plan that is representable by an intermediate representation. But I agree
>>>>>>>>>>>>>>>>>>>>>>>> this gets very complicated very soon, and just having the case (1) covered
>>>>>>>>>>>>>>>>>>>>>>>> would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny
>>>>>>>>>>>>>>>>>>>>>>>> Chow <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular
>>>>>>>>>>>>>>>>>>>>>>>> SQL UDF can be used to build a parameterized view.  So, there's definitely
>>>>>>>>>>>>>>>>>>>>>>>> a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about
>>>>>>>>>>>>>>>>>>>>>>>> what is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the
>>>>>>>>>>>>>>>>>>>>>>>> user whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's
>>>>>>>>>>>>>>>>>>>>>>>> references are pretty much from (1) and I think those have more analogy to
>>>>>>>>>>>>>>>>>>>>>>>> views due to their SQL nature. Agree (2) is not practical to maintain by
>>>>>>>>>>>>>>>>>>>>>>>> Iceberg, but I think Ajantha's use cases are around (1), and may be worth
>>>>>>>>>>>>>>>>>>>>>>>> evaluating.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post
>>>>>>>>>>>>>>>>>>>>>>>> the proposal, but I think this would be a very difficult area to tackle
>>>>>>>>>>>>>>>>>>>>>>>> across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports
>>>>>>>>>>>>>>>>>>>>>>>> SQL representations of UDFs (similar to views as shared by the reference
>>>>>>>>>>>>>>>>>>>>>>>> links above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for
>>>>>>>>>>>>>>>>>>>>>>>> your input.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft
>>>>>>>>>>>>>>>>>>>>>>>> spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack
>>>>>>>>>>>>>>>>>>>>>>>> Ye <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a
>>>>>>>>>>>>>>>>>>>>>>>> common set of functions across engines, I don't see how that is practical
>>>>>>>>>>>>>>>>>>>>>>>> when those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views?
>>>>>>>>>>>>>>>>>>>>>>>> I feel we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is
>>>>>>>>>>>>>>>>>>>>>>>> a good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we
>>>>>>>>>>>>>>>>>>>>>>>> have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM
>>>>>>>>>>>>>>>>>>>>>>>> Robert Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and
>>>>>>>>>>>>>>>>>>>>>>>> portable and "non-centralized" as views are. The same performance concerns
>>>>>>>>>>>>>>>>>>>>>>>> apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base
>>>>>>>>>>>>>>>>>>>>>>>> upon which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should
>>>>>>>>>>>>>>>>>>>>>>>> also try to tackle the idea to make views portable, which is conceptually
>>>>>>>>>>>>>>>>>>>>>>>> not that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a
>>>>>>>>>>>>>>>>>>>>>>>> negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a
>>>>>>>>>>>>>>>>>>>>>>>> good idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you
>>>>>>>>>>>>>>>>>>>>>>>> post the proposal, but I think this would be a very difficult area to
>>>>>>>>>>>>>>>>>>>>>>>> tackle across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM
>>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec
>>>>>>>>>>>>>>>>>>>>>>>> addition for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to
>>>>>>>>>>>>>>>>>>>>>>>> views in that they are associated with tables, but they can accept
>>>>>>>>>>>>>>>>>>>>>>>> arguments and produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio,
>>>>>>>>>>>>>>>>>>>>>>>> Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can
>>>>>>>>>>>>>>>>>>>>>>>> enable
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the
>>>>>>>>>>>>>>>>>>>>>>>> engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this
>>>>>>>>>>>>>>>>>>>>>>>> feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting
>>>>>>>>>>>>>>>>>>>>>>>> a specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Ajantha Bhat <aj...@gmail.com>]

Thanks to everyone who joined the sync.
Here is the meeting recording:
https://drive.google.com/file/d/1L5S6nb-C_pzBwFlClwO_sG1AVBA_ROKo/view

Summary:
We have discussed how to define function identifiers (should also handle
function overloading). Ryan suggested that we should check how Spark does
it. We can refer to functions using an identifier and then bind the
different signatures to it. So that access policies can be applied per
identifier. This is also linked to how we want to version the functions
when overloading is supported.

I will check more about this and update the proposal doc.

Please check/subscribe to the dev events calendar for the next meeting link
(Aug 11).

- Ajantha

On Sun, Jul 27, 2025 at 10:46 PM Kevin Liu <ke...@apache.org> wrote:

> Hi Ajantha,
>
> I see that the UDF Sync is scheduled in the "Iceberg Dev Events" calendar for
> tomorrow 7/28 at 9AM PT. I missed the last one, but i'll be at this one.
>
> Best,
> Kevin Liu
>
> On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
>> Hey everyone,
>>
>> No one joined the sync today. I came to know that Yufei is on holiday,
>> and Ryan and others couldn't make it, similar to the last sync. It seems
>> Yufei might have forgotten to transfer meeting ownership as well, as new
>> members needed admin approval and couldn't join automatically this week.
>> Also, I can understand it is summer holiday season for many.
>>
>> I've updated the function signature schema and other open points. I
>> believe we're very close to the final version of the spec. A meeting is
>> indeed necessary to finalize this, but we don't have to wait for it to
>> finish the review process. We had many meetings on this in the past
>> already. So, please review the document at your earliest convenience. If we
>> agree on the spec by next week, I can raise a PR.
>>
>> - Ajantha
>>
>> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> I’d propose to move the field `properties` from a top level field to a
>>> field inside “version” along with a representation, so that properties are
>>> versioned. A property like “deterministic” could change along with
>>> representation over time. For example, we need to change “deterministic”
>>> from true to false in case of adding a non-deterministic SQL
>>> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
>>> be safe.
>>>
>>> That said, it's still an open question whether we need any non-versioned
>>> properties. We can introduce them later if a use case arises.
>>>
>>> Yufei
>>>
>>>
>>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Thanks for the summary, Ajantha!
>>>>
>>>> I’d prefer to keep the signature list separate from the representation
>>>> history. Here are reasons:
>>>>
>>>>    1. Each version still enforces a single signature. Although the
>>>>    signatures array is global to the UDF, each version references just one
>>>>    signature ID. Rollbacks to historical versions remain safe.
>>>>    2. We’ve separated the less frequently changing component
>>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>>    file size.
>>>>    3. Since signatures use Iceberg data types, they should remain
>>>>    unaffected by multi-dialect representation differences.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <aj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks to everyone who joined the sync.
>>>>> Here is the meeting recording:
>>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>>
>>>>> Summary:
>>>>> We have discussed the action items from the last sync (*see Appendix
>>>>> C* in the proposal doc)
>>>>>
>>>>>    - Function overloading: Supported by few of the engines and in the
>>>>>    roadmaps of many engines. Iceberg will support it. We will maintain the
>>>>>    `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>>>    containing the function argument's type list). And all operations like
>>>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>>    - Secure UDF: If we store it as a property in a bag, we need to
>>>>>    standardize the property name. Iceberg encryption may be orthogonal to this
>>>>>    discussion.
>>>>>    - UDF with multi statement and procedural bodies are supported by
>>>>>    some engines. Iceberg will support it. Store the body as it is while
>>>>>    creating function by the engine.
>>>>>
>>>>> new discussions around
>>>>>
>>>>>    - Standardizing the property names (deterministic, secure).
>>>>>    - About the rename function.
>>>>>    - Replace function. To check upto what level replace is supported
>>>>>    (considering function overloading) .
>>>>>    - Signature should be associated with representation?
>>>>>
>>>>>    I think we are close on the spec. Please review the proposal
>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>>    .
>>>>>
>>>>> Details for next Iceberg UDF sync:
>>>>>
>>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>> Google Meet joining info
>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>
>>>>> - Ajantha
>>>>>
>>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <aj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Can it be handled by Iceberg encryption? If the whole metadata is
>>>>>> encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>>>> discuss more on the sync today.
>>>>>>
>>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, hiding the definition and disabling pushdown are required.We
>>>>>>> will need a named key(e.g., secure) somewhere, no matter if it is a top
>>>>>>> level property or a key as a part of the UDF properties. So that both UDF
>>>>>>> creator and consumer can recognize it.
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks for the extra detail. What do you think the spec would
>>>>>>>> require? Would it require hiding the UDF definition from users and require
>>>>>>>> specific pushdown cases be disabled? The use cases seem valid, but I'm
>>>>>>>> trying to understand the requirements this places on engines and why it
>>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>>
>>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Ryan,
>>>>>>>>>
>>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>>
>>>>>>>>>    1.
>>>>>>>>>
>>>>>>>>>    Hiding UDF Definitions: This includes concealing the UDF body
>>>>>>>>>    and details like the list of imports, some of them aren’t applicable to SQL
>>>>>>>>>    UDFs.
>>>>>>>>>    2.
>>>>>>>>>
>>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>>    3.
>>>>>>>>>
>>>>>>>>>    Preventing Data Leakage at Execution Time: For example, secure
>>>>>>>>>    UDFs may disable certain optimizations—such as predicate pushdown—to avoid
>>>>>>>>>    exposing sensitive data indirectly. [1]
>>>>>>>>>
>>>>>>>>> Given these scenarios, I agree with your point that the secure
>>>>>>>>> flag is primarily an instruction to the engine to behave differently. While
>>>>>>>>> it's largely an engine-side behavior, we still need to include this flag in
>>>>>>>>> the UDF definition to indicate whether a UDF is secure, especially
>>>>>>>>> considering the perf penalty introduced by scenario #3. We should clearly
>>>>>>>>> recommend that users avoid marking UDFs as secure unless it's truly
>>>>>>>>> necessary.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yufei, could you make the argument for supporting a "secure" UDF?
>>>>>>>>>> What use case are you addressing and what specifically changes about how
>>>>>>>>>> the UDF is handled? If the idea is to hide the UDF definition, do we need
>>>>>>>>>> to include it?
>>>>>>>>>>
>>>>>>>>>> I think this would be a signal to a "trusted engine". When the
>>>>>>>>>> engine interacts with the catalog it sends authorization information about
>>>>>>>>>> itself in addition to the user that it is acting on behalf of. That way the
>>>>>>>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>>>>>>>> shown to the user. The majority of this logic is on the REST server side,
>>>>>>>>>> and the only part that is communicated to the client is the request not to
>>>>>>>>>> show the UDF to the user, right? In that case should this be a property
>>>>>>>>>> rather than part of the definition? Even if we state that the client "must"
>>>>>>>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>>>>>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>>>>>>>> the definition isn't very meaningful.
>>>>>>>>>>
>>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>>
>>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>>>>>>>> statements run within a single transaction should be treated as an
>>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>>
>>>>>>>>>>> One more thing: should we also introduce a “secure UDF” option
>>>>>>>>>>> supported by some engines[1], so the body and any sensitive details stay
>>>>>>>>>>> hidden from callers?
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>>> Summary:
>>>>>>>>>>>>
>>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>>    - Each engine uses its own block separator, like $$ or ''
>>>>>>>>>>>>    or none. Action item was to check whether engines support multi-statement
>>>>>>>>>>>>    (transactional) UDF bodies.
>>>>>>>>>>>>    - Discussed about function overloading. Need to check
>>>>>>>>>>>>    whether these engines support function overloading for SQL UDFs. Postgres
>>>>>>>>>>>>    supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>>    - Started online spec review and discussed the
>>>>>>>>>>>>    deterministic flag and concluded that we keep the independent fields (like
>>>>>>>>>>>>    deterministic) in spec only if the majority of engines supports it. Else it
>>>>>>>>>>>>    will be passed in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>>
>>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>>
>>>>>>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>>>>>>
>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>
>>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>
>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>>
>>>>>>>>>>>>> Summary:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    We discussed including Python support; the majority agreed *not
>>>>>>>>>>>>>    to* (see recording for details).
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    No strong opposition to versioning — it will be included
>>>>>>>>>>>>>    to support change tracking and similar use cases.
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Suggestions were made to document how each catalog
>>>>>>>>>>>>>    resolves UDFs, similar to views and tables.
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    We agreed not to deviate from the existing table/view spec
>>>>>>>>>>>>>    — e.g., location will remain *required* for cross-catalog
>>>>>>>>>>>>>    compatibility.
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    We also discussed a bit about view interoperability as the
>>>>>>>>>>>>>    same things are applicable here.
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>>    Final spec will be put to review and vote once it is
>>>>>>>>>>>>>    ready.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF
>>>>>>>>>>>>>> project. Everyone’s welcome to drop in and share ideas! Here is the meeting
>>>>>>>>>>>>>> link:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the
>>>>>>>>>>>>>>> UDF proposal. We covered several key points, though some are still open for
>>>>>>>>>>>>>>> further discussion:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs
>>>>>>>>>>>>>>> at this stage? We explored the possibility of simplifying the specification
>>>>>>>>>>>>>>> by avoiding view replication, and potentially introducing versioning
>>>>>>>>>>>>>>> support later. UDTFs, being a superset of views in some ways, may not
>>>>>>>>>>>>>>> require versioning initially.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not
>>>>>>>>>>>>>>> support vararg syntax in CREATE FUNCTION, Iceberg UDFs
>>>>>>>>>>>>>>> could represent such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t
>>>>>>>>>>>>>>> support generic types (e.g., object), we can only map
>>>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a language for
>>>>>>>>>>>>>>> SQL UDFs seems promising, especially given its potential to resolve
>>>>>>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>>>>>>> be captured in the specification.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>>> I will update the proposal document with two primary UDF use
>>>>>>>>>>>>>>> cases:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The update will include corresponding syntax examples in
>>>>>>>>>>>>>>> both SQL and Python, and detail how each use case is represented in Iceberg
>>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more
>>>>>>>>>>>>>>> interested participants) to continue refining and finalizing the UDF
>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper
>>>>>>>>>>>>>>>> discussion is needed. Let's keep moving forward, especially with the
>>>>>>>>>>>>>>>> renewed interest from the community.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better understand
>>>>>>>>>>>>>>>>> the new requirement of using UDFs for fine-grained access control policies.
>>>>>>>>>>>>>>>>> This expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Given the strong community interest and support, I’d like
>>>>>>>>>>>>>>>>> to take ownership of this effort and revive the work. I'll be revisiting
>>>>>>>>>>>>>>>>> the document I proposed long back and will share an updated proposal by
>>>>>>>>>>>>>>>>> next week.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The UDF spec does not require representations to be SQL.
>>>>>>>>>>>>>>>>>> It merely does not specify (in this revision) how other representations are
>>>>>>>>>>>>>>>>>> to be written.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new type in
>>>>>>>>>>>>>>>>>> the "Representations" section).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec.
>>>>>>>>>>>>>>>>>>> It leaves a way for future versions to add different representations later,
>>>>>>>>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this
>>>>>>>>>>>>>>>>>>>>> proposal focuses on SQL-based engines, while Python-based systems often
>>>>>>>>>>>>>>>>>>>>> work with data frames. Adding imperative languages like Python would make
>>>>>>>>>>>>>>>>>>>>> this proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i
>>>>>>>>>>>>>>>>>>>>>> read
>>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to
>>>>>>>>>>>>>>>>>>>>>> share among different engines."
>>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully understand
>>>>>>>>>>>>>>>>>>>>>> yet how the proposed design addresses shareability between the engines
>>>>>>>>>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The
>>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created functions
>>>>>>>>>>>>>>>>>>>>>>> sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look
>>>>>>>>>>>>>>>>>>>>>>> like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple
>>>>>>>>>>>>>>>>>>>>>>> function signatures per verison, but it is unclear how to encode them
>>>>>>>>>>>>>>>>>>>>>>> because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a
>>>>>>>>>>>>>>>>>>>>>>> SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look
>>>>>>>>>>>>>>>>>>>>>>> at the proposal
>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get
>>>>>>>>>>>>>>>>>>>>>>> any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from
>>>>>>>>>>>>>>>>>>>>>>> Benny).
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a
>>>>>>>>>>>>>>>>>>>>>>> variation of (1) where the API is data flow/data pipeline API instead of
>>>>>>>>>>>>>>>>>>>>>>> SQL (e.g., Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long
>>>>>>>>>>>>>>>>>>>>>>> term opportunities in this case. Consider you register a Spark temp view as
>>>>>>>>>>>>>>>>>>>>>>> some sort of data frame read, then it could still be resolved to a Spark
>>>>>>>>>>>>>>>>>>>>>>> plan that is representable by an intermediate representation. But I agree
>>>>>>>>>>>>>>>>>>>>>>> this gets very complicated very soon, and just having the case (1) covered
>>>>>>>>>>>>>>>>>>>>>>> would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow
>>>>>>>>>>>>>>>>>>>>>>> <bt...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular
>>>>>>>>>>>>>>>>>>>>>>> SQL UDF can be used to build a parameterized view.  So, there's definitely
>>>>>>>>>>>>>>>>>>>>>>> a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa
>>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what
>>>>>>>>>>>>>>>>>>>>>>> is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the
>>>>>>>>>>>>>>>>>>>>>>> user whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references
>>>>>>>>>>>>>>>>>>>>>>> are pretty much from (1) and I think those have more analogy to views due
>>>>>>>>>>>>>>>>>>>>>>> to their SQL nature. Agree (2) is not practical to maintain by Iceberg, but
>>>>>>>>>>>>>>>>>>>>>>> I think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post
>>>>>>>>>>>>>>>>>>>>>>> the proposal, but I think this would be a very difficult area to tackle
>>>>>>>>>>>>>>>>>>>>>>> across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for
>>>>>>>>>>>>>>>>>>>>>>> your input.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft
>>>>>>>>>>>>>>>>>>>>>>> spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye
>>>>>>>>>>>>>>>>>>>>>>> <ye...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a
>>>>>>>>>>>>>>>>>>>>>>> common set of functions across engines, I don't see how that is practical
>>>>>>>>>>>>>>>>>>>>>>> when those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views?
>>>>>>>>>>>>>>>>>>>>>>> I feel we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is
>>>>>>>>>>>>>>>>>>>>>>> a good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we
>>>>>>>>>>>>>>>>>>>>>>> have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert
>>>>>>>>>>>>>>>>>>>>>>> Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and
>>>>>>>>>>>>>>>>>>>>>>> portable and "non-centralized" as views are. The same performance concerns
>>>>>>>>>>>>>>>>>>>>>>> apply to views as well.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base
>>>>>>>>>>>>>>>>>>>>>>> upon which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also
>>>>>>>>>>>>>>>>>>>>>>> try to tackle the idea to make views portable, which is conceptually not
>>>>>>>>>>>>>>>>>>>>>>> that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a
>>>>>>>>>>>>>>>>>>>>>>> negative touch to the idea of having UDFs in Iceberg, especially not in
>>>>>>>>>>>>>>>>>>>>>>> this early stage.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a
>>>>>>>>>>>>>>>>>>>>>>> good idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post
>>>>>>>>>>>>>>>>>>>>>>> the proposal, but I think this would be a very difficult area to tackle
>>>>>>>>>>>>>>>>>>>>>>> across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM
>>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition
>>>>>>>>>>>>>>>>>>>>>>> for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to
>>>>>>>>>>>>>>>>>>>>>>> views in that they are associated with tables, but they can accept
>>>>>>>>>>>>>>>>>>>>>>> arguments and produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio,
>>>>>>>>>>>>>>>>>>>>>>> Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can
>>>>>>>>>>>>>>>>>>>>>>> enable
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the
>>>>>>>>>>>>>>>>>>>>>>> engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this
>>>>>>>>>>>>>>>>>>>>>>> feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Kevin Liu <ke...@apache.org>]

Hi Ajantha,

I see that the UDF Sync is scheduled in the "Iceberg Dev Events" calendar for
tomorrow 7/28 at 9AM PT. I missed the last one, but i'll be at this one.

Best,
Kevin Liu

On Mon, Jul 14, 2025 at 9:22 AM Ajantha Bhat <aj...@gmail.com> wrote:

> Hey everyone,
>
> No one joined the sync today. I came to know that Yufei is on holiday, and
> Ryan and others couldn't make it, similar to the last sync. It seems Yufei
> might have forgotten to transfer meeting ownership as well, as new members
> needed admin approval and couldn't join automatically this week. Also, I
> can understand it is summer holiday season for many.
>
> I've updated the function signature schema and other open points. I
> believe we're very close to the final version of the spec. A meeting is
> indeed necessary to finalize this, but we don't have to wait for it to
> finish the review process. We had many meetings on this in the past
> already. So, please review the document at your earliest convenience. If we
> agree on the spec by next week, I can raise a PR.
>
> - Ajantha
>
> On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> I’d propose to move the field `properties` from a top level field to a
>> field inside “version” along with a representation, so that properties are
>> versioned. A property like “deterministic” could change along with
>> representation over time. For example, we need to change “deterministic”
>> from true to false in case of adding a non-deterministic SQL
>> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
>> be safe.
>>
>> That said, it's still an open question whether we need any non-versioned
>> properties. We can introduce them later if a use case arises.
>>
>> Yufei
>>
>>
>> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Thanks for the summary, Ajantha!
>>>
>>> I’d prefer to keep the signature list separate from the representation
>>> history. Here are reasons:
>>>
>>>    1. Each version still enforces a single signature. Although the
>>>    signatures array is global to the UDF, each version references just one
>>>    signature ID. Rollbacks to historical versions remain safe.
>>>    2. We’ve separated the less frequently changing component
>>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>>    file size.
>>>    3. Since signatures use Iceberg data types, they should remain
>>>    unaffected by multi-dialect representation differences.
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <aj...@gmail.com>
>>> wrote:
>>>
>>>> Thanks to everyone who joined the sync.
>>>> Here is the meeting recording:
>>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>>
>>>> Summary:
>>>> We have discussed the action items from the last sync (*see Appendix C* in
>>>> the proposal doc)
>>>>
>>>>    - Function overloading: Supported by few of the engines and in the
>>>>    roadmaps of many engines. Iceberg will support it. We will maintain the
>>>>    `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>>    containing the function argument's type list). And all operations like
>>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>>    - Secure UDF: If we store it as a property in a bag, we need to
>>>>    standardize the property name. Iceberg encryption may be orthogonal to this
>>>>    discussion.
>>>>    - UDF with multi statement and procedural bodies are supported by
>>>>    some engines. Iceberg will support it. Store the body as it is while
>>>>    creating function by the engine.
>>>>
>>>> new discussions around
>>>>
>>>>    - Standardizing the property names (deterministic, secure).
>>>>    - About the rename function.
>>>>    - Replace function. To check upto what level replace is supported
>>>>    (considering function overloading) .
>>>>    - Signature should be associated with representation?
>>>>
>>>>    I think we are close on the spec. Please review the proposal
>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>>    .
>>>>
>>>> Details for next Iceberg UDF sync:
>>>>
>>>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>> Google Meet joining info
>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>
>>>> - Ajantha
>>>>
>>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <aj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Can it be handled by Iceberg encryption? If the whole metadata is
>>>>> encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>>> discuss more on the sync today.
>>>>>
>>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Yes, hiding the definition and disabling pushdown are required.We
>>>>>> will need a named key(e.g., secure) somewhere, no matter if it is a top
>>>>>> level property or a key as a part of the UDF properties. So that both UDF
>>>>>> creator and consumer can recognize it.
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks for the extra detail. What do you think the spec would
>>>>>>> require? Would it require hiding the UDF definition from users and require
>>>>>>> specific pushdown cases be disabled? The use cases seem valid, but I'm
>>>>>>> trying to understand the requirements this places on engines and why it
>>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>>
>>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Ryan,
>>>>>>>>
>>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>>
>>>>>>>>    1.
>>>>>>>>
>>>>>>>>    Hiding UDF Definitions: This includes concealing the UDF body
>>>>>>>>    and details like the list of imports, some of them aren’t applicable to SQL
>>>>>>>>    UDFs.
>>>>>>>>    2.
>>>>>>>>
>>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>>    3.
>>>>>>>>
>>>>>>>>    Preventing Data Leakage at Execution Time: For example, secure
>>>>>>>>    UDFs may disable certain optimizations—such as predicate pushdown—to avoid
>>>>>>>>    exposing sensitive data indirectly. [1]
>>>>>>>>
>>>>>>>> Given these scenarios, I agree with your point that the secure
>>>>>>>> flag is primarily an instruction to the engine to behave differently. While
>>>>>>>> it's largely an engine-side behavior, we still need to include this flag in
>>>>>>>> the UDF definition to indicate whether a UDF is secure, especially
>>>>>>>> considering the perf penalty introduced by scenario #3. We should clearly
>>>>>>>> recommend that users avoid marking UDFs as secure unless it's truly
>>>>>>>> necessary.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yufei, could you make the argument for supporting a "secure" UDF?
>>>>>>>>> What use case are you addressing and what specifically changes about how
>>>>>>>>> the UDF is handled? If the idea is to hide the UDF definition, do we need
>>>>>>>>> to include it?
>>>>>>>>>
>>>>>>>>> I think this would be a signal to a "trusted engine". When the
>>>>>>>>> engine interacts with the catalog it sends authorization information about
>>>>>>>>> itself in addition to the user that it is acting on behalf of. That way the
>>>>>>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>>>>>>> shown to the user. The majority of this logic is on the REST server side,
>>>>>>>>> and the only part that is communicated to the client is the request not to
>>>>>>>>> show the UDF to the user, right? In that case should this be a property
>>>>>>>>> rather than part of the definition? Even if we state that the client "must"
>>>>>>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>>>>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>>>>>>> the definition isn't very meaningful.
>>>>>>>>>
>>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>>
>>>>>>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>>>>>>> statements run within a single transaction should be treated as an
>>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>>
>>>>>>>>>> One more thing: should we also introduce a “secure UDF” option
>>>>>>>>>> supported by some engines[1], so the body and any sensitive details stay
>>>>>>>>>> hidden from callers?
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>>> Summary:
>>>>>>>>>>>
>>>>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>>    - Each engine uses its own block separator, like $$ or '' or
>>>>>>>>>>>    none. Action item was to check whether engines support multi-statement
>>>>>>>>>>>    (transactional) UDF bodies.
>>>>>>>>>>>    - Discussed about function overloading. Need to check
>>>>>>>>>>>    whether these engines support function overloading for SQL UDFs. Postgres
>>>>>>>>>>>    supports it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>>    - Started online spec review and discussed the deterministic
>>>>>>>>>>>    flag and concluded that we keep the independent fields (like deterministic)
>>>>>>>>>>>    in spec only if the majority of engines supports it. Else it will be passed
>>>>>>>>>>>    in a property bag (engine specific). And it is the engine's
>>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>>
>>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>>
>>>>>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>>>>>
>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>
>>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>
>>>>>>>>>>> - Ajantha
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>>
>>>>>>>>>>>> Summary:
>>>>>>>>>>>>
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    We discussed including Python support; the majority agreed *not
>>>>>>>>>>>>    to* (see recording for details).
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    No strong opposition to versioning — it will be included to
>>>>>>>>>>>>    support change tracking and similar use cases.
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Suggestions were made to document how each catalog resolves
>>>>>>>>>>>>    UDFs, similar to views and tables.
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    We agreed not to deviate from the existing table/view spec
>>>>>>>>>>>>    — e.g., location will remain *required* for cross-catalog
>>>>>>>>>>>>    compatibility.
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    We also discussed a bit about view interoperability as the
>>>>>>>>>>>>    same things are applicable here.
>>>>>>>>>>>>
>>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>>    Final spec will be put to review and vote once it is ready.
>>>>>>>>>>>>
>>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>>
>>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone:
>>>>>>>>>>>> America/Los_Angeles
>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>
>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF
>>>>>>>>>>>>> project. Everyone’s welcome to drop in and share ideas! Here is the meeting
>>>>>>>>>>>>> link:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yufei
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the
>>>>>>>>>>>>>> UDF proposal. We covered several key points, though some are still open for
>>>>>>>>>>>>>> further discussion:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at
>>>>>>>>>>>>>> this stage? We explored the possibility of simplifying the specification by
>>>>>>>>>>>>>> avoiding view replication, and potentially introducing versioning support
>>>>>>>>>>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>>>>>>>>>>> versioning initially.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not
>>>>>>>>>>>>>> support vararg syntax in CREATE FUNCTION, Iceberg UDFs could
>>>>>>>>>>>>>> represent such arguments as lists when supported by the engine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t
>>>>>>>>>>>>>> support generic types (e.g., object), we can only map
>>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a language for
>>>>>>>>>>>>>> SQL UDFs seems promising, especially given its potential to resolve
>>>>>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>>>>>> be captured in the specification.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>>> I will update the proposal document with two primary UDF use
>>>>>>>>>>>>>> cases:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>>    -
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The update will include corresponding syntax examples in both
>>>>>>>>>>>>>> SQL and Python, and detail how each use case is represented in Iceberg
>>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We also plan to set up regular syncs (open to more interested
>>>>>>>>>>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper discussion
>>>>>>>>>>>>>>> is needed. Let's keep moving forward, especially with the renewed interest
>>>>>>>>>>>>>>> from the community.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I recently discussed this with Yufei to better understand
>>>>>>>>>>>>>>>> the new requirement of using UDFs for fine-grained access control policies.
>>>>>>>>>>>>>>>> This expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Given the strong community interest and support, I’d like
>>>>>>>>>>>>>>>> to take ownership of this effort and revive the work. I'll be revisiting
>>>>>>>>>>>>>>>> the document I proposed long back and will share an updated proposal by
>>>>>>>>>>>>>>>> next week.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The UDF spec does not require representations to be SQL.
>>>>>>>>>>>>>>>>> It merely does not specify (in this revision) how other representations are
>>>>>>>>>>>>>>>>> to be written.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new type in
>>>>>>>>>>>>>>>>> the "Representations" section).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It
>>>>>>>>>>>>>>>>>> leaves a way for future versions to add different representations later,
>>>>>>>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal
>>>>>>>>>>>>>>>>>>>> focuses on SQL-based engines, while Python-based systems often work with
>>>>>>>>>>>>>>>>>>>> data frames. Adding imperative languages like Python would make this
>>>>>>>>>>>>>>>>>>>> proposal more inclusive.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i
>>>>>>>>>>>>>>>>>>>>> read
>>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share
>>>>>>>>>>>>>>>>>>>>> among different engines."
>>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully understand
>>>>>>>>>>>>>>>>>>>>> yet how the proposed design addresses shareability between the engines
>>>>>>>>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The
>>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created functions
>>>>>>>>>>>>>>>>>>>>>> sharable between the engines?
>>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look
>>>>>>>>>>>>>>>>>>>>>> like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple
>>>>>>>>>>>>>>>>>>>>>> function signatures per verison, but it is unclear how to encode them
>>>>>>>>>>>>>>>>>>>>>> because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a
>>>>>>>>>>>>>>>>>>>>>> SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look
>>>>>>>>>>>>>>>>>>>>>> at the proposal
>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get
>>>>>>>>>>>>>>>>>>>>>> any review on the proposal.
>>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from
>>>>>>>>>>>>>>>>>>>>>> Benny).
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation
>>>>>>>>>>>>>>>>>>>>>> of (1) where the API is data flow/data pipeline API instead of SQL (e.g.,
>>>>>>>>>>>>>>>>>>>>>> Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long
>>>>>>>>>>>>>>>>>>>>>> term opportunities in this case. Consider you register a Spark temp view as
>>>>>>>>>>>>>>>>>>>>>> some sort of data frame read, then it could still be resolved to a Spark
>>>>>>>>>>>>>>>>>>>>>> plan that is representable by an intermediate representation. But I agree
>>>>>>>>>>>>>>>>>>>>>> this gets very complicated very soon, and just having the case (1) covered
>>>>>>>>>>>>>>>>>>>>>> would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular
>>>>>>>>>>>>>>>>>>>>>> SQL UDF can be used to build a parameterized view.  So, there's definitely
>>>>>>>>>>>>>>>>>>>>>> a lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa
>>>>>>>>>>>>>>>>>>>>>> Eldin Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what
>>>>>>>>>>>>>>>>>>>>>> is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the
>>>>>>>>>>>>>>>>>>>>>> user whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references
>>>>>>>>>>>>>>>>>>>>>> are pretty much from (1) and I think those have more analogy to views due
>>>>>>>>>>>>>>>>>>>>>> to their SQL nature. Agree (2) is not practical to maintain by Iceberg, but
>>>>>>>>>>>>>>>>>>>>>> I think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post
>>>>>>>>>>>>>>>>>>>>>> the proposal, but I think this would be a very difficult area to tackle
>>>>>>>>>>>>>>>>>>>>>> across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your
>>>>>>>>>>>>>>>>>>>>>> input.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft
>>>>>>>>>>>>>>>>>>>>>> spec (inspired by the view spec) this week to facilitate further
>>>>>>>>>>>>>>>>>>>>>> discussions.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a
>>>>>>>>>>>>>>>>>>>>>> common set of functions across engines, I don't see how that is practical
>>>>>>>>>>>>>>>>>>>>>> when those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I
>>>>>>>>>>>>>>>>>>>>>> feel we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a
>>>>>>>>>>>>>>>>>>>>>> good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we
>>>>>>>>>>>>>>>>>>>>>> have the actual proposal published.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert
>>>>>>>>>>>>>>>>>>>>>> Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and
>>>>>>>>>>>>>>>>>>>>>> portable and "non-centralized" as views are. The same performance concerns
>>>>>>>>>>>>>>>>>>>>>> apply to views as well.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base
>>>>>>>>>>>>>>>>>>>>>> upon which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also
>>>>>>>>>>>>>>>>>>>>>> try to tackle the idea to make views portable, which is conceptually not
>>>>>>>>>>>>>>>>>>>>>> that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative
>>>>>>>>>>>>>>>>>>>>>> touch to the idea of having UDFs in Iceberg, especially not in this early
>>>>>>>>>>>>>>>>>>>>>> stage.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a
>>>>>>>>>>>>>>>>>>>>>> good idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post
>>>>>>>>>>>>>>>>>>>>>> the proposal, but I think this would be a very difficult area to tackle
>>>>>>>>>>>>>>>>>>>>>> across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM
>>>>>>>>>>>>>>>>>>>>>> Ajantha Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition
>>>>>>>>>>>>>>>>>>>>>> for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to
>>>>>>>>>>>>>>>>>>>>>> views in that they are associated with tables, but they can accept
>>>>>>>>>>>>>>>>>>>>>> arguments and produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio,
>>>>>>>>>>>>>>>>>>>>>> Trino, Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the
>>>>>>>>>>>>>>>>>>>>>> engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this
>>>>>>>>>>>>>>>>>>>>>> feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Ajantha Bhat <aj...@gmail.com>]

Hey everyone,

No one joined the sync today. I came to know that Yufei is on holiday, and
Ryan and others couldn't make it, similar to the last sync. It seems Yufei
might have forgotten to transfer meeting ownership as well, as new members
needed admin approval and couldn't join automatically this week. Also, I
can understand it is summer holiday season for many.

I've updated the function signature schema and other open points. I believe
we're very close to the final version of the spec. A meeting is indeed
necessary to finalize this, but we don't have to wait for it to finish the
review process. We had many meetings on this in the past already. So,
please review the document at your earliest convenience. If we agree on the
spec by next week, I can raise a PR.

- Ajantha

On Thu, Jul 3, 2025 at 4:03 AM Yufei Gu <fl...@gmail.com> wrote:

> I’d propose to move the field `properties` from a top level field to a
> field inside “version” along with a representation, so that properties are
> versioned. A property like “deterministic” could change along with
> representation over time. For example, we need to change “deterministic”
> from true to false in case of adding a non-deterministic SQL
> expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
> be safe.
>
> That said, it's still an open question whether we need any non-versioned
> properties. We can introduce them later if a use case arises.
>
> Yufei
>
>
> On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Thanks for the summary, Ajantha!
>>
>> I’d prefer to keep the signature list separate from the representation
>> history. Here are reasons:
>>
>>    1. Each version still enforces a single signature. Although the
>>    signatures array is global to the UDF, each version references just one
>>    signature ID. Rollbacks to historical versions remain safe.
>>    2. We’ve separated the less frequently changing component
>>    (signatures) from the more dynamic one (representations) to reduce metadata
>>    file size.
>>    3. Since signatures use Iceberg data types, they should remain
>>    unaffected by multi-dialect representation differences.
>>
>> Yufei
>>
>>
>> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <aj...@gmail.com>
>> wrote:
>>
>>> Thanks to everyone who joined the sync.
>>> Here is the meeting recording:
>>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>>
>>> Summary:
>>> We have discussed the action items from the last sync (*see Appendix C* in
>>> the proposal doc)
>>>
>>>    - Function overloading: Supported by few of the engines and in the
>>>    roadmaps of many engines. Iceberg will support it. We will maintain the
>>>    `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>>    containing the function argument's type list). And all operations like
>>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>>    - Secure UDF: If we store it as a property in a bag, we need to
>>>    standardize the property name. Iceberg encryption may be orthogonal to this
>>>    discussion.
>>>    - UDF with multi statement and procedural bodies are supported by
>>>    some engines. Iceberg will support it. Store the body as it is while
>>>    creating function by the engine.
>>>
>>> new discussions around
>>>
>>>    - Standardizing the property names (deterministic, secure).
>>>    - About the rename function.
>>>    - Replace function. To check upto what level replace is supported
>>>    (considering function overloading) .
>>>    - Signature should be associated with representation?
>>>
>>>    I think we are close on the spec. Please review the proposal
>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>>    .
>>>
>>> Details for next Iceberg UDF sync:
>>>
>>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>> Google Meet joining info
>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>
>>> - Ajantha
>>>
>>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <aj...@gmail.com>
>>> wrote:
>>>
>>>> Can it be handled by Iceberg encryption? If the whole metadata is
>>>> encrypted, we don't have to worry about just hiding the UDF body? Let us
>>>> discuss more on the sync today.
>>>>
>>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Yes, hiding the definition and disabling pushdown are required.We will
>>>>> need a named key(e.g., secure) somewhere, no matter if it is a top level
>>>>> property or a key as a part of the UDF properties. So that both UDF creator
>>>>> and consumer can recognize it.
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>>
>>>>>> Thanks for the extra detail. What do you think the spec would
>>>>>> require? Would it require hiding the UDF definition from users and require
>>>>>> specific pushdown cases be disabled? The use cases seem valid, but I'm
>>>>>> trying to understand the requirements this places on engines and why it
>>>>>> needs to be part of the spec, rather than part of the properties of the UDF.
>>>>>>
>>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Ryan,
>>>>>>>
>>>>>>> Here are the main use cases for secure UDFs:
>>>>>>>
>>>>>>>    1.
>>>>>>>
>>>>>>>    Hiding UDF Definitions: This includes concealing the UDF body
>>>>>>>    and details like the list of imports, some of them aren’t applicable to SQL
>>>>>>>    UDFs.
>>>>>>>    2.
>>>>>>>
>>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>>    3.
>>>>>>>
>>>>>>>    Preventing Data Leakage at Execution Time: For example, secure
>>>>>>>    UDFs may disable certain optimizations—such as predicate pushdown—to avoid
>>>>>>>    exposing sensitive data indirectly. [1]
>>>>>>>
>>>>>>> Given these scenarios, I agree with your point that the secure flag
>>>>>>> is primarily an instruction to the engine to behave differently. While it's
>>>>>>> largely an engine-side behavior, we still need to include this flag in the
>>>>>>> UDF definition to indicate whether a UDF is secure, especially considering
>>>>>>> the perf penalty introduced by scenario #3. We should clearly recommend
>>>>>>> that users avoid marking UDFs as secure unless it's truly necessary.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Yufei, could you make the argument for supporting a "secure" UDF?
>>>>>>>> What use case are you addressing and what specifically changes about how
>>>>>>>> the UDF is handled? If the idea is to hide the UDF definition, do we need
>>>>>>>> to include it?
>>>>>>>>
>>>>>>>> I think this would be a signal to a "trusted engine". When the
>>>>>>>> engine interacts with the catalog it sends authorization information about
>>>>>>>> itself in addition to the user that it is acting on behalf of. That way the
>>>>>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>>>>>> shown to the user. The majority of this logic is on the REST server side,
>>>>>>>> and the only part that is communicated to the client is the request not to
>>>>>>>> show the UDF to the user, right? In that case should this be a property
>>>>>>>> rather than part of the definition? Even if we state that the client "must"
>>>>>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>>>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>>>>>> the definition isn't very meaningful.
>>>>>>>>
>>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>>
>>>>>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>>>>>> statements run within a single transaction should be treated as an
>>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>>
>>>>>>>>> One more thing: should we also introduce a “secure UDF” option
>>>>>>>>> supported by some engines[1], so the body and any sensitive details stay
>>>>>>>>> hidden from callers?
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>>> Summary:
>>>>>>>>>>
>>>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>>    - Each engine uses its own block separator, like $$ or '' or
>>>>>>>>>>    none. Action item was to check whether engines support multi-statement
>>>>>>>>>>    (transactional) UDF bodies.
>>>>>>>>>>    - Discussed about function overloading. Need to check whether
>>>>>>>>>>    these engines support function overloading for SQL UDFs. Postgres supports
>>>>>>>>>>    it! If yes, need to adopt the spec to handle it.
>>>>>>>>>>    - Started online spec review and discussed the deterministic
>>>>>>>>>>    flag and concluded that we keep the independent fields (like deterministic)
>>>>>>>>>>    in spec only if the majority of engines supports it. Else it will be passed
>>>>>>>>>>    in a property bag (engine specific). And it is the engine's
>>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>>
>>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>>
>>>>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>>>>
>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>
>>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>>>> Google Meet joining info
>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>
>>>>>>>>>> - Ajantha
>>>>>>>>>>
>>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>>
>>>>>>>>>>> Summary:
>>>>>>>>>>>
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    We discussed including Python support; the majority agreed *not
>>>>>>>>>>>    to* (see recording for details).
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    No strong opposition to versioning — it will be included to
>>>>>>>>>>>    support change tracking and similar use cases.
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Suggestions were made to document how each catalog resolves
>>>>>>>>>>>    UDFs, similar to views and tables.
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    We agreed not to deviate from the existing table/view spec —
>>>>>>>>>>>    e.g., location will remain *required* for cross-catalog
>>>>>>>>>>>    compatibility.
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    We also discussed a bit about view interoperability as the
>>>>>>>>>>>    same things are applicable here.
>>>>>>>>>>>
>>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>>    Final spec will be put to review and vote once it is ready.
>>>>>>>>>>>
>>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>>
>>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>
>>>>>>>>>>> - Ajantha
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>
>>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF
>>>>>>>>>>>> project. Everyone’s welcome to drop in and share ideas! Here is the meeting
>>>>>>>>>>>> link:
>>>>>>>>>>>>
>>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>>
>>>>>>>>>>>> Yufei
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the
>>>>>>>>>>>>> UDF proposal. We covered several key points, though some are still open for
>>>>>>>>>>>>> further discussion:
>>>>>>>>>>>>>
>>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at
>>>>>>>>>>>>> this stage? We explored the possibility of simplifying the specification by
>>>>>>>>>>>>> avoiding view replication, and potentially introducing versioning support
>>>>>>>>>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>>>>>>>>>> versioning initially.
>>>>>>>>>>>>>
>>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not
>>>>>>>>>>>>> support vararg syntax in CREATE FUNCTION, Iceberg UDFs could
>>>>>>>>>>>>> represent such arguments as lists when supported by the engine.
>>>>>>>>>>>>>
>>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t
>>>>>>>>>>>>> support generic types (e.g., object), we can only map
>>>>>>>>>>>>> engine-specific types to Iceberg types. As a result, generic data types
>>>>>>>>>>>>> will not be supported in the initial version.
>>>>>>>>>>>>>
>>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a language for
>>>>>>>>>>>>> SQL UDFs seems promising, especially given its potential to resolve
>>>>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>>>>> be captured in the specification.
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>>> I will update the proposal document with two primary UDF use
>>>>>>>>>>>>> cases:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>>    -
>>>>>>>>>>>>>
>>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>>
>>>>>>>>>>>>> The update will include corresponding syntax examples in both
>>>>>>>>>>>>> SQL and Python, and detail how each use case is represented in Iceberg
>>>>>>>>>>>>> metadata.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We also plan to set up regular syncs (open to more interested
>>>>>>>>>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper discussion
>>>>>>>>>>>>>> is needed. Let's keep moving forward, especially with the renewed interest
>>>>>>>>>>>>>> from the community.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> During the last catalog community sync, there was
>>>>>>>>>>>>>>> significant interest in storing UDFs in Iceberg and adding endpoints for
>>>>>>>>>>>>>>> UDF handling in the REST catalog spec.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I recently discussed this with Yufei to better understand
>>>>>>>>>>>>>>> the new requirement of using UDFs for fine-grained access control policies.
>>>>>>>>>>>>>>> This expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Given the strong community interest and support, I’d like to
>>>>>>>>>>>>>>> take ownership of this effort and revive the work. I'll be revisiting the
>>>>>>>>>>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The UDF spec does not require representations to be SQL. It
>>>>>>>>>>>>>>>> merely does not specify (in this revision) how other representations are to
>>>>>>>>>>>>>>>> be written.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>>>>>>>>>>> "Representations" section).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It
>>>>>>>>>>>>>>>>> leaves a way for future versions to add different representations later,
>>>>>>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal
>>>>>>>>>>>>>>>>>>> focuses on SQL-based engines, while Python-based systems often work with
>>>>>>>>>>>>>>>>>>> data frames. Adding imperative languages like Python would make this
>>>>>>>>>>>>>>>>>>> proposal more inclusive.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i
>>>>>>>>>>>>>>>>>>>> read
>>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share
>>>>>>>>>>>>>>>>>>>> among different engines."
>>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully understand
>>>>>>>>>>>>>>>>>>>> yet how the proposed design addresses shareability between the engines
>>>>>>>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in
>>>>>>>>>>>>>>>>>>>>> imperative code?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The
>>>>>>>>>>>>>>>>>>>>> Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created functions
>>>>>>>>>>>>>>>>>>>>> sharable between the engines?
>>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look
>>>>>>>>>>>>>>>>>>>>> like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is
>>>>>>>>>>>>>>>>>>>>> inconsistent with conventions in other specs. For example, using string IDs
>>>>>>>>>>>>>>>>>>>>> rather than an integer.
>>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec
>>>>>>>>>>>>>>>>>>>>> versioning than the Iceberg view and table specs. It requires readers to
>>>>>>>>>>>>>>>>>>>>> fail if there are any unknown fields, which prevents the spec from adding
>>>>>>>>>>>>>>>>>>>>> things that are fully backward-compatible. Other Iceberg specs only require
>>>>>>>>>>>>>>>>>>>>> a version change to introduce forward-incompatible changes and I think that
>>>>>>>>>>>>>>>>>>>>> this should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple
>>>>>>>>>>>>>>>>>>>>> function signatures per verison, but it is unclear how to encode them
>>>>>>>>>>>>>>>>>>>>> because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a
>>>>>>>>>>>>>>>>>>>>> SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look
>>>>>>>>>>>>>>>>>>>>> at the proposal
>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get
>>>>>>>>>>>>>>>>>>>>> any review on the proposal.
>>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from
>>>>>>>>>>>>>>>>>>>>> Benny).
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation
>>>>>>>>>>>>>>>>>>>>> of (1) where the API is data flow/data pipeline API instead of SQL (e.g.,
>>>>>>>>>>>>>>>>>>>>> Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long
>>>>>>>>>>>>>>>>>>>>> term opportunities in this case. Consider you register a Spark temp view as
>>>>>>>>>>>>>>>>>>>>> some sort of data frame read, then it could still be resolved to a Spark
>>>>>>>>>>>>>>>>>>>>> plan that is representable by an intermediate representation. But I agree
>>>>>>>>>>>>>>>>>>>>> this gets very complicated very soon, and just having the case (1) covered
>>>>>>>>>>>>>>>>>>>>> would already be a huge step forward.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL
>>>>>>>>>>>>>>>>>>>>> UDF can be used to build a parameterized view.  So, there's definitely a
>>>>>>>>>>>>>>>>>>>>> lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what
>>>>>>>>>>>>>>>>>>>>> is perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user
>>>>>>>>>>>>>>>>>>>>> whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references
>>>>>>>>>>>>>>>>>>>>> are pretty much from (1) and I think those have more analogy to views due
>>>>>>>>>>>>>>>>>>>>> to their SQL nature. Agree (2) is not practical to maintain by Iceberg, but
>>>>>>>>>>>>>>>>>>>>> I think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post
>>>>>>>>>>>>>>>>>>>>> the proposal, but I think this would be a very difficult area to tackle
>>>>>>>>>>>>>>>>>>>>> across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your
>>>>>>>>>>>>>>>>>>>>> input.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a
>>>>>>>>>>>>>>>>>>>>> common set of functions across engines, I don't see how that is practical
>>>>>>>>>>>>>>>>>>>>> when those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I
>>>>>>>>>>>>>>>>>>>>> feel we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a
>>>>>>>>>>>>>>>>>>>>> good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have
>>>>>>>>>>>>>>>>>>>>> the actual proposal published.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert
>>>>>>>>>>>>>>>>>>>>> Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and
>>>>>>>>>>>>>>>>>>>>> portable and "non-centralized" as views are. The same performance concerns
>>>>>>>>>>>>>>>>>>>>> apply to views as well.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base
>>>>>>>>>>>>>>>>>>>>> upon which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also
>>>>>>>>>>>>>>>>>>>>> try to tackle the idea to make views portable, which is conceptually not
>>>>>>>>>>>>>>>>>>>>> that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative
>>>>>>>>>>>>>>>>>>>>> touch to the idea of having UDFs in Iceberg, especially not in this early
>>>>>>>>>>>>>>>>>>>>> stage.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good
>>>>>>>>>>>>>>>>>>>>> idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post
>>>>>>>>>>>>>>>>>>>>> the proposal, but I think this would be a very difficult area to tackle
>>>>>>>>>>>>>>>>>>>>> across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha
>>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition
>>>>>>>>>>>>>>>>>>>>> for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to
>>>>>>>>>>>>>>>>>>>>> views in that they are associated with tables, but they can accept
>>>>>>>>>>>>>>>>>>>>> arguments and produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the
>>>>>>>>>>>>>>>>>>>>> engines. Potentially engines can understand the UDFs written by other
>>>>>>>>>>>>>>>>>>>>> engines (with the translate layer).
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this
>>>>>>>>>>>>>>>>>>>>> feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

I’d propose to move the field `properties` from a top level field to a
field inside “version” along with a representation, so that properties are
versioned. A property like “deterministic” could change along with
representation over time. For example, we need to change “deterministic”
from true to false in case of adding a non-deterministic SQL
expression/function(e.g., now()) inside an UDF. Otherwise, rollback won't
be safe.

That said, it's still an open question whether we need any non-versioned
properties. We can introduce them later if a use case arises.

Yufei


On Wed, Jul 2, 2025 at 3:06 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks for the summary, Ajantha!
>
> I’d prefer to keep the signature list separate from the representation
> history. Here are reasons:
>
>    1. Each version still enforces a single signature. Although the
>    signatures array is global to the UDF, each version references just one
>    signature ID. Rollbacks to historical versions remain safe.
>    2. We’ve separated the less frequently changing component (signatures)
>    from the more dynamic one (representations) to reduce metadata file size.
>    3. Since signatures use Iceberg data types, they should remain
>    unaffected by multi-dialect representation differences.
>
> Yufei
>
>
> On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
>> Thanks to everyone who joined the sync.
>> Here is the meeting recording:
>> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>>
>> Summary:
>> We have discussed the action items from the last sync (*see Appendix C* in
>> the proposal doc)
>>
>>    - Function overloading: Supported by few of the engines and in the
>>    roadmaps of many engines. Iceberg will support it. We will maintain the
>>    `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>>    containing the function argument's type list). And all operations like
>>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>>    - Secure UDF: If we store it as a property in a bag, we need to
>>    standardize the property name. Iceberg encryption may be orthogonal to this
>>    discussion.
>>    - UDF with multi statement and procedural bodies are supported by
>>    some engines. Iceberg will support it. Store the body as it is while
>>    creating function by the engine.
>>
>> new discussions around
>>
>>    - Standardizing the property names (deterministic, secure).
>>    - About the rename function.
>>    - Replace function. To check upto what level replace is supported
>>    (considering function overloading) .
>>    - Signature should be associated with representation?
>>
>>    I think we are close on the spec. Please review the proposal
>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>>    .
>>
>> Details for next Iceberg UDF sync:
>>
>> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>> Google Meet joining info
>> Video call link: https://meet.google.com/aui-czix-nbh
>>
>> - Ajantha
>>
>> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <aj...@gmail.com>
>> wrote:
>>
>>> Can it be handled by Iceberg encryption? If the whole metadata is
>>> encrypted, we don't have to worry about just hiding the UDF body? Let us
>>> discuss more on the sync today.
>>>
>>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Yes, hiding the definition and disabling pushdown are required.We will
>>>> need a named key(e.g., secure) somewhere, no matter if it is a top level
>>>> property or a key as a part of the UDF properties. So that both UDF creator
>>>> and consumer can recognize it.
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>
>>>>> Thanks for the extra detail. What do you think the spec would require?
>>>>> Would it require hiding the UDF definition from users and require specific
>>>>> pushdown cases be disabled? The use cases seem valid, but I'm trying to
>>>>> understand the requirements this places on engines and why it needs to be
>>>>> part of the spec, rather than part of the properties of the UDF.
>>>>>
>>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi Ryan,
>>>>>>
>>>>>> Here are the main use cases for secure UDFs:
>>>>>>
>>>>>>    1.
>>>>>>
>>>>>>    Hiding UDF Definitions: This includes concealing the UDF body and
>>>>>>    details like the list of imports, some of them aren’t applicable to SQL
>>>>>>    UDFs.
>>>>>>    2.
>>>>>>
>>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>>    3.
>>>>>>
>>>>>>    Preventing Data Leakage at Execution Time: For example, secure
>>>>>>    UDFs may disable certain optimizations—such as predicate pushdown—to avoid
>>>>>>    exposing sensitive data indirectly. [1]
>>>>>>
>>>>>> Given these scenarios, I agree with your point that the secure flag
>>>>>> is primarily an instruction to the engine to behave differently. While it's
>>>>>> largely an engine-side behavior, we still need to include this flag in the
>>>>>> UDF definition to indicate whether a UDF is secure, especially considering
>>>>>> the perf penalty introduced by scenario #3. We should clearly recommend
>>>>>> that users avoid marking UDFs as secure unless it's truly necessary.
>>>>>>
>>>>>> [1]
>>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>>>
>>>>>>> Yufei, could you make the argument for supporting a "secure" UDF?
>>>>>>> What use case are you addressing and what specifically changes about how
>>>>>>> the UDF is handled? If the idea is to hide the UDF definition, do we need
>>>>>>> to include it?
>>>>>>>
>>>>>>> I think this would be a signal to a "trusted engine". When the
>>>>>>> engine interacts with the catalog it sends authorization information about
>>>>>>> itself in addition to the user that it is acting on behalf of. That way the
>>>>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>>>>> shown to the user. The majority of this logic is on the REST server side,
>>>>>>> and the only part that is communicated to the client is the request not to
>>>>>>> show the UDF to the user, right? In that case should this be a property
>>>>>>> rather than part of the definition? Even if we state that the client "must"
>>>>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>>>>> the definition isn't very meaningful.
>>>>>>>
>>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>>
>>>>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>>>>> statements run within a single transaction should be treated as an
>>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>>
>>>>>>>> One more thing: should we also introduce a “secure UDF” option
>>>>>>>> supported by some engines[1], so the body and any sensitive details stay
>>>>>>>> hidden from callers?
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>> Here is the meeting recording:
>>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>>> Summary:
>>>>>>>>>
>>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>>    - Each engine uses its own block separator, like $$ or '' or
>>>>>>>>>    none. Action item was to check whether engines support multi-statement
>>>>>>>>>    (transactional) UDF bodies.
>>>>>>>>>    - Discussed about function overloading. Need to check whether
>>>>>>>>>    these engines support function overloading for SQL UDFs. Postgres supports
>>>>>>>>>    it! If yes, need to adopt the spec to handle it.
>>>>>>>>>    - Started online spec review and discussed the deterministic
>>>>>>>>>    flag and concluded that we keep the independent fields (like deterministic)
>>>>>>>>>    in spec only if the majority of engines supports it. Else it will be passed
>>>>>>>>>    in a property bag (engine specific). And it is the engine's
>>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>>
>>>>>>>>> Feel free to review the current proposal document here
>>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>>
>>>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>>>
>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>
>>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>>> Google Meet joining info
>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>
>>>>>>>>> - Ajantha
>>>>>>>>>
>>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>>> Here is the meeting recording:
>>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>>
>>>>>>>>>> Summary:
>>>>>>>>>>
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    We discussed including Python support; the majority agreed *not
>>>>>>>>>>    to* (see recording for details).
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    No strong opposition to versioning — it will be included to
>>>>>>>>>>    support change tracking and similar use cases.
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Suggestions were made to document how each catalog resolves
>>>>>>>>>>    UDFs, similar to views and tables.
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    We agreed not to deviate from the existing table/view spec —
>>>>>>>>>>    e.g., location will remain *required* for cross-catalog
>>>>>>>>>>    compatibility.
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    We also discussed a bit about view interoperability as the
>>>>>>>>>>    same things are applicable here.
>>>>>>>>>>
>>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>>    Final spec will be put to review and vote once it is ready.
>>>>>>>>>>
>>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>>
>>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>>>> Google Meet joining info
>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>
>>>>>>>>>> - Ajantha
>>>>>>>>>>
>>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi folks,
>>>>>>>>>>>
>>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF
>>>>>>>>>>> project. Everyone’s welcome to drop in and share ideas! Here is the meeting
>>>>>>>>>>> link:
>>>>>>>>>>>
>>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>>> Google Meet joining info
>>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>>
>>>>>>>>>>> Yufei
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>>
>>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>>>>>>>>>>> proposal. We covered several key points, though some are still open for
>>>>>>>>>>>> further discussion:
>>>>>>>>>>>>
>>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at
>>>>>>>>>>>> this stage? We explored the possibility of simplifying the specification by
>>>>>>>>>>>> avoiding view replication, and potentially introducing versioning support
>>>>>>>>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>>>>>>>>> versioning initially.
>>>>>>>>>>>>
>>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not support
>>>>>>>>>>>> vararg syntax in CREATE FUNCTION, Iceberg UDFs could represent
>>>>>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>>>>>
>>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support
>>>>>>>>>>>> generic types (e.g., object), we can only map engine-specific
>>>>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>>>>> supported in the initial version.
>>>>>>>>>>>>
>>>>>>>>>>>> d) *Python Support*: Incorporating Python as a language for
>>>>>>>>>>>> SQL UDFs seems promising, especially given its potential to resolve
>>>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>>>> be captured in the specification.
>>>>>>>>>>>>
>>>>>>>>>>>> *Next Steps*
>>>>>>>>>>>> I will update the proposal document with two primary UDF use
>>>>>>>>>>>> cases:
>>>>>>>>>>>>
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>>
>>>>>>>>>>>> The update will include corresponding syntax examples in both
>>>>>>>>>>>> SQL and Python, and detail how each use case is represented in Iceberg
>>>>>>>>>>>> metadata.
>>>>>>>>>>>>
>>>>>>>>>>>> We also plan to set up regular syncs (open to more interested
>>>>>>>>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper discussion
>>>>>>>>>>>>> is needed. Let's keep moving forward, especially with the renewed interest
>>>>>>>>>>>>> from the community.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> During the last catalog community sync, there was significant
>>>>>>>>>>>>>> interest in storing UDFs in Iceberg and adding endpoints for UDF handling
>>>>>>>>>>>>>> in the REST catalog spec.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I recently discussed this with Yufei to better understand the
>>>>>>>>>>>>>> new requirement of using UDFs for fine-grained access control policies.
>>>>>>>>>>>>>> This expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Given the strong community interest and support, I’d like to
>>>>>>>>>>>>>> take ownership of this effort and revive the work. I'll be revisiting the
>>>>>>>>>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The UDF spec does not require representations to be SQL. It
>>>>>>>>>>>>>>> merely does not specify (in this revision) how other representations are to
>>>>>>>>>>>>>>> be written.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>>>>>>>>>> "Representations" section).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It
>>>>>>>>>>>>>>>> leaves a way for future versions to add different representations later,
>>>>>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal
>>>>>>>>>>>>>>>>>> focuses on SQL-based engines, while Python-based systems often work with
>>>>>>>>>>>>>>>>>> data frames. Adding imperative languages like Python would make this
>>>>>>>>>>>>>>>>>> proposal more inclusive.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i
>>>>>>>>>>>>>>>>>>> read
>>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share
>>>>>>>>>>>>>>>>>>> among different engines."
>>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully understand
>>>>>>>>>>>>>>>>>>> yet how the proposed design addresses shareability between the engines
>>>>>>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created
>>>>>>>>>>>>>>>>>>>> functions shareable
>>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in imperative
>>>>>>>>>>>>>>>>>>>> code?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The
>>>>>>>>>>>>>>>>>>>> Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created functions
>>>>>>>>>>>>>>>>>>>> sharable between the engines?
>>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look
>>>>>>>>>>>>>>>>>>>> like in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is inconsistent
>>>>>>>>>>>>>>>>>>>> with conventions in other specs. For example, using string IDs rather than
>>>>>>>>>>>>>>>>>>>> an integer.
>>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec versioning
>>>>>>>>>>>>>>>>>>>> than the Iceberg view and table specs. It requires readers to fail if there
>>>>>>>>>>>>>>>>>>>> are any unknown fields, which prevents the spec from adding things that are
>>>>>>>>>>>>>>>>>>>> fully backward-compatible. Other Iceberg specs only require a version
>>>>>>>>>>>>>>>>>>>> change to introduce forward-incompatible changes and I think that this
>>>>>>>>>>>>>>>>>>>> should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple
>>>>>>>>>>>>>>>>>>>> function signatures per verison, but it is unclear how to encode them
>>>>>>>>>>>>>>>>>>>> because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a
>>>>>>>>>>>>>>>>>>>> SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look at
>>>>>>>>>>>>>>>>>>>> the proposal
>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin
>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any
>>>>>>>>>>>>>>>>>>>> review on the proposal.
>>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from
>>>>>>>>>>>>>>>>>>>> Benny).
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the
>>>>>>>>>>>>>>>>>>>> decisions and how we want to implement it.
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation
>>>>>>>>>>>>>>>>>>>> of (1) where the API is data flow/data pipeline API instead of SQL (e.g.,
>>>>>>>>>>>>>>>>>>>> Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL
>>>>>>>>>>>>>>>>>>>> UDF can be used to build a parameterized view.  So, there's definitely a
>>>>>>>>>>>>>>>>>>>> lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user
>>>>>>>>>>>>>>>>>>>> whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references
>>>>>>>>>>>>>>>>>>>> are pretty much from (1) and I think those have more analogy to views due
>>>>>>>>>>>>>>>>>>>> to their SQL nature. Agree (2) is not practical to maintain by Iceberg, but
>>>>>>>>>>>>>>>>>>>> I think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha
>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your
>>>>>>>>>>>>>>>>>>>> input.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a
>>>>>>>>>>>>>>>>>>>> common set of functions across engines, I don't see how that is practical
>>>>>>>>>>>>>>>>>>>> when those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I
>>>>>>>>>>>>>>>>>>>> feel we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a
>>>>>>>>>>>>>>>>>>>> good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have
>>>>>>>>>>>>>>>>>>>> the actual proposal published.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert
>>>>>>>>>>>>>>>>>>>> Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable
>>>>>>>>>>>>>>>>>>>> and "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>>>>>>>>>>> views as well.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon
>>>>>>>>>>>>>>>>>>>> which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also
>>>>>>>>>>>>>>>>>>>> try to tackle the idea to make views portable, which is conceptually not
>>>>>>>>>>>>>>>>>>>> that much different from portable UDFs.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative
>>>>>>>>>>>>>>>>>>>> touch to the idea of having UDFs in Iceberg, especially not in this early
>>>>>>>>>>>>>>>>>>>> stage.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good
>>>>>>>>>>>>>>>>>>>> idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post
>>>>>>>>>>>>>>>>>>>> the proposal, but I think this would be a very difficult area to tackle
>>>>>>>>>>>>>>>>>>>> across engines, languages, and memory models without having a huge
>>>>>>>>>>>>>>>>>>>> performance penalty.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha
>>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition
>>>>>>>>>>>>>>>>>>>> for storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to
>>>>>>>>>>>>>>>>>>>> views in that they are associated with tables, but they can accept
>>>>>>>>>>>>>>>>>>>> arguments and produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>>>>>>>>>>> the translate layer).
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this
>>>>>>>>>>>>>>>>>>>> feature into Iceberg would be a valuable addition, and we're eager to
>>>>>>>>>>>>>>>>>>>> collaborate with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Thanks for the summary, Ajantha!

I’d prefer to keep the signature list separate from the representation
history. Here are reasons:

   1. Each version still enforces a single signature. Although the
   signatures array is global to the UDF, each version references just one
   signature ID. Rollbacks to historical versions remain safe.
   2. We’ve separated the less frequently changing component (signatures)
   from the more dynamic one (representations) to reduce metadata file size.
   3. Since signatures use Iceberg data types, they should remain
   unaffected by multi-dialect representation differences.

Yufei


On Mon, Jun 30, 2025 at 11:28 AM Ajantha Bhat <aj...@gmail.com> wrote:

> Thanks to everyone who joined the sync.
> Here is the meeting recording:
> https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing
>
> Summary:
> We have discussed the action items from the last sync (*see Appendix C* in
> the proposal doc)
>
>    - Function overloading: Supported by few of the engines and in the
>    roadmaps of many engines. Iceberg will support it. We will maintain the
>    `FunctionIdentifier` (extends `TableIdentifer` but also have a member
>    containing the function argument's type list). And all operations like
>    load, rename, list, create and drop are based on `FunctionIdentifier`.
>    - Secure UDF: If we store it as a property in a bag, we need to
>    standardize the property name. Iceberg encryption may be orthogonal to this
>    discussion.
>    - UDF with multi statement and procedural bodies are supported by some
>    engines. Iceberg will support it. Store the body as it is while creating
>    function by the engine.
>
> new discussions around
>
>    - Standardizing the property names (deterministic, secure).
>    - About the rename function.
>    - Replace function. To check upto what level replace is supported
>    (considering function overloading) .
>    - Signature should be associated with representation?
>
>    I think we are close on the spec. Please review the proposal
>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
>    .
>
> Details for next Iceberg UDF sync:
>
> *Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
> Google Meet joining info
> Video call link: https://meet.google.com/aui-czix-nbh
>
> - Ajantha
>
> On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
>> Can it be handled by Iceberg encryption? If the whole metadata is
>> encrypted, we don't have to worry about just hiding the UDF body? Let us
>> discuss more on the sync today.
>>
>> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Yes, hiding the definition and disabling pushdown are required.We will
>>> need a named key(e.g., secure) somewhere, no matter if it is a top level
>>> property or a key as a part of the UDF properties. So that both UDF creator
>>> and consumer can recognize it.
>>>
>>> Yufei
>>>
>>>
>>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com> wrote:
>>>
>>>> Thanks for the extra detail. What do you think the spec would require?
>>>> Would it require hiding the UDF definition from users and require specific
>>>> pushdown cases be disabled? The use cases seem valid, but I'm trying to
>>>> understand the requirements this places on engines and why it needs to be
>>>> part of the spec, rather than part of the properties of the UDF.
>>>>
>>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi Ryan,
>>>>>
>>>>> Here are the main use cases for secure UDFs:
>>>>>
>>>>>    1.
>>>>>
>>>>>    Hiding UDF Definitions: This includes concealing the UDF body and
>>>>>    details like the list of imports, some of them aren’t applicable to SQL
>>>>>    UDFs.
>>>>>    2.
>>>>>
>>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>>    3.
>>>>>
>>>>>    Preventing Data Leakage at Execution Time: For example, secure
>>>>>    UDFs may disable certain optimizations—such as predicate pushdown—to avoid
>>>>>    exposing sensitive data indirectly. [1]
>>>>>
>>>>> Given these scenarios, I agree with your point that the secure flag
>>>>> is primarily an instruction to the engine to behave differently. While it's
>>>>> largely an engine-side behavior, we still need to include this flag in the
>>>>> UDF definition to indicate whether a UDF is secure, especially considering
>>>>> the perf penalty introduced by scenario #3. We should clearly recommend
>>>>> that users avoid marking UDFs as secure unless it's truly necessary.
>>>>>
>>>>> [1]
>>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>>
>>>>>> Yufei, could you make the argument for supporting a "secure" UDF?
>>>>>> What use case are you addressing and what specifically changes about how
>>>>>> the UDF is handled? If the idea is to hide the UDF definition, do we need
>>>>>> to include it?
>>>>>>
>>>>>> I think this would be a signal to a "trusted engine". When the engine
>>>>>> interacts with the catalog it sends authorization information about itself
>>>>>> in addition to the user that it is acting on behalf of. That way the
>>>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>>>> shown to the user. The majority of this logic is on the REST server side,
>>>>>> and the only part that is communicated to the client is the request not to
>>>>>> show the UDF to the user, right? In that case should this be a property
>>>>>> rather than part of the definition? Even if we state that the client "must"
>>>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>>>> the definition isn't very meaningful.
>>>>>>
>>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the summary, Ajantha!
>>>>>>>
>>>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>>>> statements run within a single transaction should be treated as an
>>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>>
>>>>>>> One more thing: should we also introduce a “secure UDF” option
>>>>>>> supported by some engines[1], so the body and any sensitive details stay
>>>>>>> hidden from callers?
>>>>>>>
>>>>>>> [1]
>>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>> Here is the meeting recording:
>>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>>> Summary:
>>>>>>>>
>>>>>>>>    - We have gone through the SQL UDF syntax supported by
>>>>>>>>    different engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>>    - Each engine uses its own block separator, like $$ or '' or
>>>>>>>>    none. Action item was to check whether engines support multi-statement
>>>>>>>>    (transactional) UDF bodies.
>>>>>>>>    - Discussed about function overloading. Need to check whether
>>>>>>>>    these engines support function overloading for SQL UDFs. Postgres supports
>>>>>>>>    it! If yes, need to adopt the spec to handle it.
>>>>>>>>    - Started online spec review and discussed the deterministic
>>>>>>>>    flag and concluded that we keep the independent fields (like deterministic)
>>>>>>>>    in spec only if the majority of engines supports it. Else it will be passed
>>>>>>>>    in a property bag (engine specific). And it is the engine's
>>>>>>>>    responsibility to honor those optional properties.
>>>>>>>>
>>>>>>>> Feel free to review the current proposal document here
>>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>>
>>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>>
>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>
>>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>> Google Meet joining info
>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>
>>>>>>>> - Ajantha
>>>>>>>>
>>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>>> Here is the meeting recording:
>>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>>
>>>>>>>>> Summary:
>>>>>>>>>
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    We discussed including Python support; the majority agreed *not
>>>>>>>>>    to* (see recording for details).
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    No strong opposition to versioning — it will be included to
>>>>>>>>>    support change tracking and similar use cases.
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Suggestions were made to document how each catalog resolves
>>>>>>>>>    UDFs, similar to views and tables.
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    We agreed not to deviate from the existing table/view spec —
>>>>>>>>>    e.g., location will remain *required* for cross-catalog
>>>>>>>>>    compatibility.
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    We also discussed a bit about view interoperability as the
>>>>>>>>>    same things are applicable here.
>>>>>>>>>
>>>>>>>>>    Feel free to review the proposal document
>>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>>    Final spec will be put to review and vote once it is ready.
>>>>>>>>>
>>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>>
>>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>>> Google Meet joining info
>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>
>>>>>>>>> - Ajantha
>>>>>>>>>
>>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF
>>>>>>>>>> project. Everyone’s welcome to drop in and share ideas! Here is the meeting
>>>>>>>>>> link:
>>>>>>>>>>
>>>>>>>>>> Iceberg UDF sync
>>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>>> Google Meet joining info
>>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>>
>>>>>>>>>> Yufei
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Update on the progress.
>>>>>>>>>>>
>>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>>>>>>>>>> proposal. We covered several key points, though some are still open for
>>>>>>>>>>> further discussion:
>>>>>>>>>>>
>>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at
>>>>>>>>>>> this stage? We explored the possibility of simplifying the specification by
>>>>>>>>>>> avoiding view replication, and potentially introducing versioning support
>>>>>>>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>>>>>>>> versioning initially.
>>>>>>>>>>>
>>>>>>>>>>> b) *VarArgs Support*: While some query engines may not support
>>>>>>>>>>> vararg syntax in CREATE FUNCTION, Iceberg UDFs could represent
>>>>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>>>>
>>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support
>>>>>>>>>>> generic types (e.g., object), we can only map engine-specific
>>>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>>>> supported in the initial version.
>>>>>>>>>>>
>>>>>>>>>>> d) *Python Support*: Incorporating Python as a language for SQL
>>>>>>>>>>> UDFs seems promising, especially given its potential to resolve
>>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>>> be captured in the specification.
>>>>>>>>>>>
>>>>>>>>>>> *Next Steps*
>>>>>>>>>>> I will update the proposal document with two primary UDF use
>>>>>>>>>>> cases:
>>>>>>>>>>>
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>>
>>>>>>>>>>> The update will include corresponding syntax examples in both
>>>>>>>>>>> SQL and Python, and detail how each use case is represented in Iceberg
>>>>>>>>>>> metadata.
>>>>>>>>>>>
>>>>>>>>>>> We also plan to set up regular syncs (open to more interested
>>>>>>>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>>>>>>>> - Ajantha
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper discussion is
>>>>>>>>>>>> needed. Let's keep moving forward, especially with the renewed interest
>>>>>>>>>>>> from the community.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> During the last catalog community sync, there was significant
>>>>>>>>>>>>> interest in storing UDFs in Iceberg and adding endpoints for UDF handling
>>>>>>>>>>>>> in the REST catalog spec.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I recently discussed this with Yufei to better understand the
>>>>>>>>>>>>> new requirement of using UDFs for fine-grained access control policies.
>>>>>>>>>>>>> This expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Given the strong community interest and support, I’d like to
>>>>>>>>>>>>> take ownership of this effort and revive the work. I'll be revisiting the
>>>>>>>>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>>>>>>>>> week.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The UDF spec does not require representations to be SQL. It
>>>>>>>>>>>>>> merely does not specify (in this revision) how other representations are to
>>>>>>>>>>>>>> be written.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>>>>>>>>> "Representations" section).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It
>>>>>>>>>>>>>>> leaves a way for future versions to add different representations later,
>>>>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal
>>>>>>>>>>>>>>>>> focuses on SQL-based engines, while Python-based systems often work with
>>>>>>>>>>>>>>>>> data frames. Adding imperative languages like Python would make this
>>>>>>>>>>>>>>>>> proposal more inclusive.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i
>>>>>>>>>>>>>>>>>> read
>>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share
>>>>>>>>>>>>>>>>>> among different engines."
>>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully understand yet
>>>>>>>>>>>>>>>>>> how the proposed design addresses shareability between the engines though.
>>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created functions
>>>>>>>>>>>>>>>>>>> shareable
>>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in imperative
>>>>>>>>>>>>>>>>>>> code?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The
>>>>>>>>>>>>>>>>>>> Iceberg UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created functions
>>>>>>>>>>>>>>>>>>> sharable between the engines?
>>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like
>>>>>>>>>>>>>>>>>>> in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added
>>>>>>>>>>>>>>>>>>> comments. I think it would be helpful to also have a design doc that covers
>>>>>>>>>>>>>>>>>>> the choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>>> >> There are a few instances where this is inconsistent
>>>>>>>>>>>>>>>>>>> with conventions in other specs. For example, using string IDs rather than
>>>>>>>>>>>>>>>>>>> an integer.
>>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec versioning
>>>>>>>>>>>>>>>>>>> than the Iceberg view and table specs. It requires readers to fail if there
>>>>>>>>>>>>>>>>>>> are any unknown fields, which prevents the spec from adding things that are
>>>>>>>>>>>>>>>>>>> fully backward-compatible. Other Iceberg specs only require a version
>>>>>>>>>>>>>>>>>>> change to introduce forward-incompatible changes and I think that this
>>>>>>>>>>>>>>>>>>> should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple
>>>>>>>>>>>>>>>>>>> function signatures per verison, but it is unclear how to encode them
>>>>>>>>>>>>>>>>>>> because a version is associated with a single function signature.
>>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a
>>>>>>>>>>>>>>>>>>> SELECT statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look at
>>>>>>>>>>>>>>>>>>> the proposal
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa
>>>>>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any
>>>>>>>>>>>>>>>>>>> review on the proposal.
>>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from
>>>>>>>>>>>>>>>>>>> Benny).
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions
>>>>>>>>>>>>>>>>>>> and how we want to implement it.
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of
>>>>>>>>>>>>>>>>>>> (1) where the API is data flow/data pipeline API instead of SQL (e.g.,
>>>>>>>>>>>>>>>>>>> Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL
>>>>>>>>>>>>>>>>>>> UDF can be used to build a parameterized view.  So, there's definitely a
>>>>>>>>>>>>>>>>>>> lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin
>>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user
>>>>>>>>>>>>>>>>>>> whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are
>>>>>>>>>>>>>>>>>>> pretty much from (1) and I think those have more analogy to views due to
>>>>>>>>>>>>>>>>>>> their SQL nature. Agree (2) is not practical to maintain by Iceberg, but I
>>>>>>>>>>>>>>>>>>> think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat
>>>>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your
>>>>>>>>>>>>>>>>>>> input.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common
>>>>>>>>>>>>>>>>>>> set of functions across engines, I don't see how that is practical when
>>>>>>>>>>>>>>>>>>> those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I
>>>>>>>>>>>>>>>>>>> feel we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a
>>>>>>>>>>>>>>>>>>> good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have
>>>>>>>>>>>>>>>>>>> the actual proposal published.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert
>>>>>>>>>>>>>>>>>>> Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable
>>>>>>>>>>>>>>>>>>> and "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>>>>>>>>>> views as well.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon
>>>>>>>>>>>>>>>>>>> which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try
>>>>>>>>>>>>>>>>>>> to tackle the idea to make views portable, which is conceptually not that
>>>>>>>>>>>>>>>>>>> much different from portable UDFs.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative
>>>>>>>>>>>>>>>>>>> touch to the idea of having UDFs in Iceberg, especially not in this early
>>>>>>>>>>>>>>>>>>> stage.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good
>>>>>>>>>>>>>>>>>>> idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha
>>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for
>>>>>>>>>>>>>>>>>>> storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to
>>>>>>>>>>>>>>>>>>> views in that they are associated with tables, but they can accept
>>>>>>>>>>>>>>>>>>> arguments and produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>>>>>>>>>> the translate layer).
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature
>>>>>>>>>>>>>>>>>>> into Iceberg would be a valuable addition, and we're eager to collaborate
>>>>>>>>>>>>>>>>>>> with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Ajantha Bhat <aj...@gmail.com>]

Thanks to everyone who joined the sync.
Here is the meeting recording:
https://drive.google.com/file/d/1FcOSbHo9ZIVeZXdUlmoG42o-chB7Q15P/view?usp=sharing

Summary:
We have discussed the action items from the last sync (*see Appendix C* in
the proposal doc)

   - Function overloading: Supported by few of the engines and in the
   roadmaps of many engines. Iceberg will support it. We will maintain the
   `FunctionIdentifier` (extends `TableIdentifer` but also have a member
   containing the function argument's type list). And all operations like
   load, rename, list, create and drop are based on `FunctionIdentifier`.
   - Secure UDF: If we store it as a property in a bag, we need to
   standardize the property name. Iceberg encryption may be orthogonal to this
   discussion.
   - UDF with multi statement and procedural bodies are supported by some
   engines. Iceberg will support it. Store the body as it is while creating
   function by the engine.

new discussions around

   - Standardizing the property names (deterministic, secure).
   - About the rename function.
   - Replace function. To check upto what level replace is supported
   (considering function overloading) .
   - Signature should be associated with representation?

   I think we are close on the spec. Please review the proposal
   <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>
   .

Details for next Iceberg UDF sync:

*Monday, July 14 · 9:00 – 10:00am*Time zone: America/Los_Angeles
Google Meet joining info
Video call link: https://meet.google.com/aui-czix-nbh

- Ajantha

On Mon, Jun 30, 2025 at 9:27 PM Ajantha Bhat <aj...@gmail.com> wrote:

> Can it be handled by Iceberg encryption? If the whole metadata is
> encrypted, we don't have to worry about just hiding the UDF body? Let us
> discuss more on the sync today.
>
> On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Yes, hiding the definition and disabling pushdown are required.We will
>> need a named key(e.g., secure) somewhere, no matter if it is a top level
>> property or a key as a part of the UDF properties. So that both UDF creator
>> and consumer can recognize it.
>>
>> Yufei
>>
>>
>> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com> wrote:
>>
>>> Thanks for the extra detail. What do you think the spec would require?
>>> Would it require hiding the UDF definition from users and require specific
>>> pushdown cases be disabled? The use cases seem valid, but I'm trying to
>>> understand the requirements this places on engines and why it needs to be
>>> part of the spec, rather than part of the properties of the UDF.
>>>
>>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi Ryan,
>>>>
>>>> Here are the main use cases for secure UDFs:
>>>>
>>>>    1.
>>>>
>>>>    Hiding UDF Definitions: This includes concealing the UDF body and
>>>>    details like the list of imports, some of them aren’t applicable to SQL
>>>>    UDFs.
>>>>    2.
>>>>
>>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>>    3.
>>>>
>>>>    Preventing Data Leakage at Execution Time: For example, secure UDFs
>>>>    may disable certain optimizations—such as predicate pushdown—to avoid
>>>>    exposing sensitive data indirectly. [1]
>>>>
>>>> Given these scenarios, I agree with your point that the secure flag is
>>>> primarily an instruction to the engine to behave differently. While it's
>>>> largely an engine-side behavior, we still need to include this flag in the
>>>> UDF definition to indicate whether a UDF is secure, especially considering
>>>> the perf penalty introduced by scenario #3. We should clearly recommend
>>>> that users avoid marking UDFs as secure unless it's truly necessary.
>>>>
>>>> [1]
>>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>>> Yufei
>>>>
>>>>
>>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>
>>>>> Yufei, could you make the argument for supporting a "secure" UDF? What
>>>>> use case are you addressing and what specifically changes about how the UDF
>>>>> is handled? If the idea is to hide the UDF definition, do we need to
>>>>> include it?
>>>>>
>>>>> I think this would be a signal to a "trusted engine". When the engine
>>>>> interacts with the catalog it sends authorization information about itself
>>>>> in addition to the user that it is acting on behalf of. That way the
>>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>>> shown to the user. The majority of this logic is on the REST server side,
>>>>> and the only part that is communicated to the client is the request not to
>>>>> show the UDF to the user, right? In that case should this be a property
>>>>> rather than part of the definition? Even if we state that the client "must"
>>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>>> the definition isn't very meaningful.
>>>>>
>>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Thanks for the summary, Ajantha!
>>>>>>
>>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>>> statements run within a single transaction should be treated as an
>>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>>
>>>>>> One more thing: should we also introduce a “secure UDF” option
>>>>>> supported by some engines[1], so the body and any sensitive details stay
>>>>>> hidden from callers?
>>>>>>
>>>>>> [1]
>>>>>> https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <aj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks to everyone who joined the sync.
>>>>>>> Here is the meeting recording:
>>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>>> Summary:
>>>>>>>
>>>>>>>    - We have gone through the SQL UDF syntax supported by different
>>>>>>>    engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>>    - Each engine uses its own block separator, like $$ or '' or
>>>>>>>    none. Action item was to check whether engines support multi-statement
>>>>>>>    (transactional) UDF bodies.
>>>>>>>    - Discussed about function overloading. Need to check whether
>>>>>>>    these engines support function overloading for SQL UDFs. Postgres supports
>>>>>>>    it! If yes, need to adopt the spec to handle it.
>>>>>>>    - Started online spec review and discussed the deterministic
>>>>>>>    flag and concluded that we keep the independent fields (like deterministic)
>>>>>>>    in spec only if the majority of engines supports it. Else it will be passed
>>>>>>>    in a property bag (engine specific). And it is the engine's
>>>>>>>    responsibility to honor those optional properties.
>>>>>>>
>>>>>>> Feel free to review the current proposal document here
>>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>>
>>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>>
>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>
>>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>> Google Meet joining info
>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>
>>>>>>> - Ajantha
>>>>>>>
>>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks to everyone who joined the sync.
>>>>>>>> Here is the meeting recording:
>>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>>
>>>>>>>> Summary:
>>>>>>>>
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    We discussed including Python support; the majority agreed *not
>>>>>>>>    to* (see recording for details).
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    No strong opposition to versioning — it will be included to
>>>>>>>>    support change tracking and similar use cases.
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Suggestions were made to document how each catalog resolves
>>>>>>>>    UDFs, similar to views and tables.
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    We agreed not to deviate from the existing table/view spec —
>>>>>>>>    e.g., location will remain *required* for cross-catalog
>>>>>>>>    compatibility.
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    We also discussed a bit about view interoperability as the same
>>>>>>>>    things are applicable here.
>>>>>>>>
>>>>>>>>    Feel free to review the proposal document
>>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>>    Final spec will be put to review and vote once it is ready.
>>>>>>>>
>>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>>
>>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>>> Google Meet joining info
>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>
>>>>>>>> - Ajantha
>>>>>>>>
>>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF
>>>>>>>>> project. Everyone’s welcome to drop in and share ideas! Here is the meeting
>>>>>>>>> link:
>>>>>>>>>
>>>>>>>>> Iceberg UDF sync
>>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>>> Google Meet joining info
>>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>>
>>>>>>>>> Yufei
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Update on the progress.
>>>>>>>>>>
>>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>>>>>>>>> proposal. We covered several key points, though some are still open for
>>>>>>>>>> further discussion:
>>>>>>>>>>
>>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at
>>>>>>>>>> this stage? We explored the possibility of simplifying the specification by
>>>>>>>>>> avoiding view replication, and potentially introducing versioning support
>>>>>>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>>>>>>> versioning initially.
>>>>>>>>>>
>>>>>>>>>> b) *VarArgs Support*: While some query engines may not support
>>>>>>>>>> vararg syntax in CREATE FUNCTION, Iceberg UDFs could represent
>>>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>>>
>>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support
>>>>>>>>>> generic types (e.g., object), we can only map engine-specific
>>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>>> supported in the initial version.
>>>>>>>>>>
>>>>>>>>>> d) *Python Support*: Incorporating Python as a language for SQL
>>>>>>>>>> UDFs seems promising, especially given its potential to resolve
>>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>>> be captured in the specification.
>>>>>>>>>>
>>>>>>>>>> *Next Steps*
>>>>>>>>>> I will update the proposal document with two primary UDF use
>>>>>>>>>> cases:
>>>>>>>>>>
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Policy exchange between engines
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>>
>>>>>>>>>> The update will include corresponding syntax examples in both SQL
>>>>>>>>>> and Python, and detail how each use case is represented in Iceberg metadata.
>>>>>>>>>>
>>>>>>>>>> We also plan to set up regular syncs (open to more interested
>>>>>>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>>>>>>> - Ajantha
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>
>>>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>>
>>>>>>>>>>> I'm happy to schedule a separate sync if a deeper discussion is
>>>>>>>>>>> needed. Let's keep moving forward, especially with the renewed interest
>>>>>>>>>>> from the community.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> During the last catalog community sync, there was significant
>>>>>>>>>>>> interest in storing UDFs in Iceberg and adding endpoints for UDF handling
>>>>>>>>>>>> in the REST catalog spec.
>>>>>>>>>>>>
>>>>>>>>>>>> I recently discussed this with Yufei to better understand the
>>>>>>>>>>>> new requirement of using UDFs for fine-grained access control policies.
>>>>>>>>>>>> This expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>>>>
>>>>>>>>>>>> Given the strong community interest and support, I’d like to
>>>>>>>>>>>> take ownership of this effort and revive the work. I'll be revisiting the
>>>>>>>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>>>>>>>> week.
>>>>>>>>>>>>
>>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The UDF spec does not require representations to be SQL. It
>>>>>>>>>>>>> merely does not specify (in this revision) how other representations are to
>>>>>>>>>>>>> be written.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>>>>>>>> "Representations" section).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It
>>>>>>>>>>>>>> leaves a way for future versions to add different representations later,
>>>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal
>>>>>>>>>>>>>>>> focuses on SQL-based engines, while Python-based systems often work with
>>>>>>>>>>>>>>>> data frames. Adding imperative languages like Python would make this
>>>>>>>>>>>>>>>> proposal more inclusive.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share
>>>>>>>>>>>>>>>>> among different engines."
>>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>>> I agree with this statement. I don't fully understand yet
>>>>>>>>>>>>>>>>> how the proposed design addresses shareability between the engines though.
>>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created functions
>>>>>>>>>>>>>>>>>> shareable
>>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in imperative
>>>>>>>>>>>>>>>>>> code?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg
>>>>>>>>>>>>>>>>>> UDFs are an interesting idea!
>>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created functions
>>>>>>>>>>>>>>>>>> sharable between the engines?
>>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like
>>>>>>>>>>>>>>>>>> in e..g Spark or Trino?
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added comments.
>>>>>>>>>>>>>>>>>> I think it would be helpful to also have a design doc that covers the
>>>>>>>>>>>>>>>>>> choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>>> >> There are a few instances where this is inconsistent
>>>>>>>>>>>>>>>>>> with conventions in other specs. For example, using string IDs rather than
>>>>>>>>>>>>>>>>>> an integer.
>>>>>>>>>>>>>>>>>> >> This uses a very different model for spec versioning
>>>>>>>>>>>>>>>>>> than the Iceberg view and table specs. It requires readers to fail if there
>>>>>>>>>>>>>>>>>> are any unknown fields, which prevents the spec from adding things that are
>>>>>>>>>>>>>>>>>> fully backward-compatible. Other Iceberg specs only require a version
>>>>>>>>>>>>>>>>>> change to introduce forward-incompatible changes and I think that this
>>>>>>>>>>>>>>>>>> should do the same to avoid confusion.
>>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>>>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>>>>>>>>>>> version is associated with a single function signature.
>>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating
>>>>>>>>>>>>>>>>>> functions across engines, so this doesn’t show that the metadata proposed
>>>>>>>>>>>>>>>>>> is sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>>>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review
>>>>>>>>>>>>>>>>>> comments, I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look at
>>>>>>>>>>>>>>>>>> the proposal
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any
>>>>>>>>>>>>>>>>>> review on the proposal.
>>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions
>>>>>>>>>>>>>>>>>> and how we want to implement it.
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin
>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of
>>>>>>>>>>>>>>>>>> (1) where the API is data flow/data pipeline API instead of SQL (e.g.,
>>>>>>>>>>>>>>>>>> Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL
>>>>>>>>>>>>>>>>>> UDF can be used to build a parameterized view.  So, there's definitely a
>>>>>>>>>>>>>>>>>> lot in common between UDFs and views.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin
>>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user
>>>>>>>>>>>>>>>>>> whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative
>>>>>>>>>>>>>>>>>> function according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are
>>>>>>>>>>>>>>>>>> pretty much from (1) and I think those have more analogy to views due to
>>>>>>>>>>>>>>>>>> their SQL nature. Agree (2) is not practical to maintain by Iceberg, but I
>>>>>>>>>>>>>>>>>> think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your
>>>>>>>>>>>>>>>>>> input.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common
>>>>>>>>>>>>>>>>>> set of functions across engines, I don't see how that is practical when
>>>>>>>>>>>>>>>>>> those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I
>>>>>>>>>>>>>>>>>> feel we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a
>>>>>>>>>>>>>>>>>> good idea, even just a multi-dialect one like view, and that can allow
>>>>>>>>>>>>>>>>>> engines to for example parse a view SQL, and when a function referenced
>>>>>>>>>>>>>>>>>> cannot be resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have
>>>>>>>>>>>>>>>>>> the actual proposal published.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert
>>>>>>>>>>>>>>>>>> Stupp <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable
>>>>>>>>>>>>>>>>>> and "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>>>>>>>>> views as well.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon
>>>>>>>>>>>>>>>>>> which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try
>>>>>>>>>>>>>>>>>> to tackle the idea to make views portable, which is conceptually not that
>>>>>>>>>>>>>>>>>> much different from portable UDFs.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative
>>>>>>>>>>>>>>>>>> touch to the idea of having UDFs in Iceberg, especially not in this early
>>>>>>>>>>>>>>>>>> stage.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good
>>>>>>>>>>>>>>>>>> idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha
>>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for
>>>>>>>>>>>>>>>>>> storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views
>>>>>>>>>>>>>>>>>> in that they are associated with tables, but they can accept arguments and
>>>>>>>>>>>>>>>>>> produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>>>>>>>>> the translate layer).
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature
>>>>>>>>>>>>>>>>>> into Iceberg would be a valuable addition, and we're eager to collaborate
>>>>>>>>>>>>>>>>>> with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Ajantha Bhat <aj...@gmail.com>]

Can it be handled by Iceberg encryption? If the whole metadata is
encrypted, we don't have to worry about just hiding the UDF body? Let us
discuss more on the sync today.

On Mon, Jun 30, 2025 at 9:22 PM Yufei Gu <fl...@gmail.com> wrote:

> Yes, hiding the definition and disabling pushdown are required.We will
> need a named key(e.g., secure) somewhere, no matter if it is a top level
> property or a key as a part of the UDF properties. So that both UDF creator
> and consumer can recognize it.
>
> Yufei
>
>
> On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com> wrote:
>
>> Thanks for the extra detail. What do you think the spec would require?
>> Would it require hiding the UDF definition from users and require specific
>> pushdown cases be disabled? The use cases seem valid, but I'm trying to
>> understand the requirements this places on engines and why it needs to be
>> part of the spec, rather than part of the properties of the UDF.
>>
>> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi Ryan,
>>>
>>> Here are the main use cases for secure UDFs:
>>>
>>>    1.
>>>
>>>    Hiding UDF Definitions: This includes concealing the UDF body and
>>>    details like the list of imports, some of them aren’t applicable to SQL
>>>    UDFs.
>>>    2.
>>>
>>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>>    3.
>>>
>>>    Preventing Data Leakage at Execution Time: For example, secure UDFs
>>>    may disable certain optimizations—such as predicate pushdown—to avoid
>>>    exposing sensitive data indirectly. [1]
>>>
>>> Given these scenarios, I agree with your point that the secure flag is
>>> primarily an instruction to the engine to behave differently. While it's
>>> largely an engine-side behavior, we still need to include this flag in the
>>> UDF definition to indicate whether a UDF is secure, especially considering
>>> the perf penalty introduced by scenario #3. We should clearly recommend
>>> that users avoid marking UDFs as secure unless it's truly necessary.
>>>
>>> [1]
>>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>>> Yufei
>>>
>>>
>>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com> wrote:
>>>
>>>> Yufei, could you make the argument for supporting a "secure" UDF? What
>>>> use case are you addressing and what specifically changes about how the UDF
>>>> is handled? If the idea is to hide the UDF definition, do we need to
>>>> include it?
>>>>
>>>> I think this would be a signal to a "trusted engine". When the engine
>>>> interacts with the catalog it sends authorization information about itself
>>>> in addition to the user that it is acting on behalf of. That way the
>>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>>> shown to the user. The majority of this logic is on the REST server side,
>>>> and the only part that is communicated to the client is the request not to
>>>> show the UDF to the user, right? In that case should this be a property
>>>> rather than part of the definition? Even if we state that the client "must"
>>>> suppress the UDF definition, it's really just a request. Only trusted
>>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>>> the definition isn't very meaningful.
>>>>
>>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Thanks for the summary, Ajantha!
>>>>>
>>>>> Multi-statement UDFs are definitely useful, but whether those
>>>>> statements run within a single transaction should be treated as an
>>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>>> declares itself transactional, the engine may or may not enforce it.
>>>>>
>>>>> One more thing: should we also introduce a “secure UDF” option
>>>>> supported by some engines[1], so the body and any sensitive details stay
>>>>> hidden from callers?
>>>>>
>>>>> [1] https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <aj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks to everyone who joined the sync.
>>>>>> Here is the meeting recording:
>>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>>> Summary:
>>>>>>
>>>>>>    - We have gone through the SQL UDF syntax supported by different
>>>>>>    engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>>    - Each engine uses its own block separator, like $$ or '' or
>>>>>>    none. Action item was to check whether engines support multi-statement
>>>>>>    (transactional) UDF bodies.
>>>>>>    - Discussed about function overloading. Need to check whether
>>>>>>    these engines support function overloading for SQL UDFs. Postgres supports
>>>>>>    it! If yes, need to adopt the spec to handle it.
>>>>>>    - Started online spec review and discussed the deterministic flag
>>>>>>    and concluded that we keep the independent fields (like deterministic) in
>>>>>>    spec only if the majority of engines supports it. Else it will be passed in
>>>>>>    a property bag (engine specific). And it is the engine's responsibility to
>>>>>>    honor those optional properties.
>>>>>>
>>>>>> Feel free to review the current proposal document here
>>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>>
>>>>>> Final spec will be put to review and vote once it is ready.
>>>>>>
>>>>>> Details for next Iceberg UDF sync:
>>>>>>
>>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>> Google Meet joining info
>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>
>>>>>> - Ajantha
>>>>>>
>>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks to everyone who joined the sync.
>>>>>>> Here is the meeting recording:
>>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>>
>>>>>>> Summary:
>>>>>>>
>>>>>>>    -
>>>>>>>
>>>>>>>    We discussed including Python support; the majority agreed *not
>>>>>>>    to* (see recording for details).
>>>>>>>    -
>>>>>>>
>>>>>>>    No strong opposition to versioning — it will be included to
>>>>>>>    support change tracking and similar use cases.
>>>>>>>    -
>>>>>>>
>>>>>>>    Suggestions were made to document how each catalog resolves
>>>>>>>    UDFs, similar to views and tables.
>>>>>>>    -
>>>>>>>
>>>>>>>    We agreed not to deviate from the existing table/view spec —
>>>>>>>    e.g., location will remain *required* for cross-catalog
>>>>>>>    compatibility.
>>>>>>>    -
>>>>>>>
>>>>>>>    We also discussed a bit about view interoperability as the same
>>>>>>>    things are applicable here.
>>>>>>>
>>>>>>>    Feel free to review the proposal document
>>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>>    Final spec will be put to review and vote once it is ready.
>>>>>>>
>>>>>>> Details for next Iceberg UDF sync:
>>>>>>>
>>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>>> Google Meet joining info
>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>
>>>>>>> - Ajantha
>>>>>>>
>>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF
>>>>>>>> project. Everyone’s welcome to drop in and share ideas! Here is the meeting
>>>>>>>> link:
>>>>>>>>
>>>>>>>> Iceberg UDF sync
>>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>>> Time zone: America/Los_Angeles
>>>>>>>> Google Meet joining info
>>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>>
>>>>>>>> Yufei
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Update on the progress.
>>>>>>>>>
>>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>>>>>>>> proposal. We covered several key points, though some are still open for
>>>>>>>>> further discussion:
>>>>>>>>>
>>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at this
>>>>>>>>> stage? We explored the possibility of simplifying the specification by
>>>>>>>>> avoiding view replication, and potentially introducing versioning support
>>>>>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>>>>>> versioning initially.
>>>>>>>>>
>>>>>>>>> b) *VarArgs Support*: While some query engines may not support
>>>>>>>>> vararg syntax in CREATE FUNCTION, Iceberg UDFs could represent
>>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>>
>>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support
>>>>>>>>> generic types (e.g., object), we can only map engine-specific
>>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>>> supported in the initial version.
>>>>>>>>>
>>>>>>>>> d) *Python Support*: Incorporating Python as a language for SQL
>>>>>>>>> UDFs seems promising, especially given its potential to resolve
>>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>>> be captured in the specification.
>>>>>>>>>
>>>>>>>>> *Next Steps*
>>>>>>>>> I will update the proposal document with two primary UDF use cases:
>>>>>>>>>
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Policy exchange between engines
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>>
>>>>>>>>> The update will include corresponding syntax examples in both SQL
>>>>>>>>> and Python, and detail how each use case is represented in Iceberg metadata.
>>>>>>>>>
>>>>>>>>> We also plan to set up regular syncs (open to more interested
>>>>>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>>>>>> - Ajantha
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>>
>>>>>>>>>> I'm happy to schedule a separate sync if a deeper discussion is
>>>>>>>>>> needed. Let's keep moving forward, especially with the renewed interest
>>>>>>>>>> from the community.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey everyone,
>>>>>>>>>>>
>>>>>>>>>>> During the last catalog community sync, there was significant
>>>>>>>>>>> interest in storing UDFs in Iceberg and adding endpoints for UDF handling
>>>>>>>>>>> in the REST catalog spec.
>>>>>>>>>>>
>>>>>>>>>>> I recently discussed this with Yufei to better understand the
>>>>>>>>>>> new requirement of using UDFs for fine-grained access control policies.
>>>>>>>>>>> This expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>>>
>>>>>>>>>>> Given the strong community interest and support, I’d like to
>>>>>>>>>>> take ownership of this effort and revive the work. I'll be revisiting the
>>>>>>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>>>>>>> week.
>>>>>>>>>>>
>>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>>> - Ajantha
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The UDF spec does not require representations to be SQL. It
>>>>>>>>>>>> merely does not specify (in this revision) how other representations are to
>>>>>>>>>>>> be written.
>>>>>>>>>>>>
>>>>>>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>>>>>>> "Representations" section).
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It
>>>>>>>>>>>>> leaves a way for future versions to add different representations later,
>>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal
>>>>>>>>>>>>>>> focuses on SQL-based engines, while Python-based systems often work with
>>>>>>>>>>>>>>> data frames. Adding imperative languages like Python would make this
>>>>>>>>>>>>>>> proposal more inclusive.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share
>>>>>>>>>>>>>>>> among different engines."
>>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>>> I agree with this statement. I don't fully understand yet
>>>>>>>>>>>>>>>> how the proposed design addresses shareability between the engines though.
>>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created functions
>>>>>>>>>>>>>>>>> shareable
>>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in imperative
>>>>>>>>>>>>>>>>> code?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg
>>>>>>>>>>>>>>>>> UDFs are an interesting idea!
>>>>>>>>>>>>>>>>> > Is there a plan to make the user-created functions
>>>>>>>>>>>>>>>>> sharable between the engines?
>>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like
>>>>>>>>>>>>>>>>> in e..g Spark or Trino?
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> I just looked through the proposal and added comments.
>>>>>>>>>>>>>>>>> I think it would be helpful to also have a design doc that covers the
>>>>>>>>>>>>>>>>> choices from the draft spec. For instance, the choice to enumerate all
>>>>>>>>>>>>>>>>> possible function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function
>>>>>>>>>>>>>>>>> signatures is limiting. It would be nice to see a discussion of the
>>>>>>>>>>>>>>>>> trade-offs and a rationale for the choice. I think it would also be very
>>>>>>>>>>>>>>>>> helpful to have a few representative use cases for this included in the
>>>>>>>>>>>>>>>>> doc. That way the proposal can demonstrate that it solves those use cases
>>>>>>>>>>>>>>>>> with reasonable trade-offs.
>>>>>>>>>>>>>>>>> >> There are a few instances where this is inconsistent
>>>>>>>>>>>>>>>>> with conventions in other specs. For example, using string IDs rather than
>>>>>>>>>>>>>>>>> an integer.
>>>>>>>>>>>>>>>>> >> This uses a very different model for spec versioning
>>>>>>>>>>>>>>>>> than the Iceberg view and table specs. It requires readers to fail if there
>>>>>>>>>>>>>>>>> are any unknown fields, which prevents the spec from adding things that are
>>>>>>>>>>>>>>>>> fully backward-compatible. Other Iceberg specs only require a version
>>>>>>>>>>>>>>>>> change to introduce forward-incompatible changes and I think that this
>>>>>>>>>>>>>>>>> should do the same to avoid confusion.
>>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>>>>>>>>>> version is associated with a single function signature.
>>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating functions
>>>>>>>>>>>>>>>>> across engines, so this doesn’t show that the metadata proposed is
>>>>>>>>>>>>>>>>> sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review comments,
>>>>>>>>>>>>>>>>> I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look at
>>>>>>>>>>>>>>>>> the proposal
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any
>>>>>>>>>>>>>>>>> review on the proposal.
>>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions
>>>>>>>>>>>>>>>>> and how we want to implement it.
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin
>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of
>>>>>>>>>>>>>>>>> (1) where the API is data flow/data pipeline API instead of SQL (e.g.,
>>>>>>>>>>>>>>>>> Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF
>>>>>>>>>>>>>>>>> can be used to build a parameterized view.  So, there's definitely a lot in
>>>>>>>>>>>>>>>>> common between UDFs and views.
>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin
>>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user
>>>>>>>>>>>>>>>>> whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are
>>>>>>>>>>>>>>>>> pretty much from (1) and I think those have more analogy to views due to
>>>>>>>>>>>>>>>>> their SQL nature. Agree (2) is not practical to maintain by Iceberg, but I
>>>>>>>>>>>>>>>>> think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your
>>>>>>>>>>>>>>>>> input.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common
>>>>>>>>>>>>>>>>> set of functions across engines, I don't see how that is practical when
>>>>>>>>>>>>>>>>> those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel
>>>>>>>>>>>>>>>>> we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good
>>>>>>>>>>>>>>>>> idea, even just a multi-dialect one like view, and that can allow engines
>>>>>>>>>>>>>>>>> to for example parse a view SQL, and when a function referenced cannot be
>>>>>>>>>>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the
>>>>>>>>>>>>>>>>> actual proposal published.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp
>>>>>>>>>>>>>>>>> <sn...@snazy.de> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable
>>>>>>>>>>>>>>>>> and "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>>>>>>>> views as well.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon
>>>>>>>>>>>>>>>>> which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try
>>>>>>>>>>>>>>>>> to tackle the idea to make views portable, which is conceptually not that
>>>>>>>>>>>>>>>>> much different from portable UDFs.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative
>>>>>>>>>>>>>>>>> touch to the idea of having UDFs in Iceberg, especially not in this early
>>>>>>>>>>>>>>>>> stage.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good
>>>>>>>>>>>>>>>>> idea to add UDFs tracked by Iceberg catalogs. I think that Iceberg
>>>>>>>>>>>>>>>>> primarily deals with things that are centralized, like tables of data.
>>>>>>>>>>>>>>>>> While it would be great to have a common set of functions across engines, I
>>>>>>>>>>>>>>>>> don't see how that is practical when those engines are implemented so
>>>>>>>>>>>>>>>>> differently. Plugging in code -- and especially custom user-supplied code
>>>>>>>>>>>>>>>>> -- seems inherently specialized to me and should be part of the engines'
>>>>>>>>>>>>>>>>> design.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha
>>>>>>>>>>>>>>>>> Bhat <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the
>>>>>>>>>>>>>>>>> community interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for
>>>>>>>>>>>>>>>>> storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views
>>>>>>>>>>>>>>>>> in that they are associated with tables, but they can accept arguments and
>>>>>>>>>>>>>>>>> produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>>>>>>>> the translate layer).
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature
>>>>>>>>>>>>>>>>> into Iceberg would be a valuable addition, and we're eager to collaborate
>>>>>>>>>>>>>>>>> with the community to develop a UDF specification.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>> Databricks
>>>>>>>>>>>>>
>>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Yes, hiding the definition and disabling pushdown are required.We will need
a named key(e.g., secure) somewhere, no matter if it is a top level
property or a key as a part of the UDF properties. So that both UDF creator
and consumer can recognize it.

Yufei


On Thu, Jun 26, 2025 at 4:27 PM Ryan Blue <rd...@gmail.com> wrote:

> Thanks for the extra detail. What do you think the spec would require?
> Would it require hiding the UDF definition from users and require specific
> pushdown cases be disabled? The use cases seem valid, but I'm trying to
> understand the requirements this places on engines and why it needs to be
> part of the spec, rather than part of the properties of the UDF.
>
> On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi Ryan,
>>
>> Here are the main use cases for secure UDFs:
>>
>>    1.
>>
>>    Hiding UDF Definitions: This includes concealing the UDF body and
>>    details like the list of imports, some of them aren’t applicable to SQL
>>    UDFs.
>>    2.
>>
>>    Sandboxed Execution: Ensuring the UDF runs in an isolated
>>    environment. Again, this typically doesn’t apply to SQL UDFs.
>>    3.
>>
>>    Preventing Data Leakage at Execution Time: For example, secure UDFs
>>    may disable certain optimizations—such as predicate pushdown—to avoid
>>    exposing sensitive data indirectly. [1]
>>
>> Given these scenarios, I agree with your point that the secure flag is
>> primarily an instruction to the engine to behave differently. While it's
>> largely an engine-side behavior, we still need to include this flag in the
>> UDF definition to indicate whether a UDF is secure, especially considering
>> the perf penalty introduced by scenario #3. We should clearly recommend
>> that users avoid marking UDFs as secure unless it's truly necessary.
>>
>> [1]
>> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
>> Yufei
>>
>>
>> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com> wrote:
>>
>>> Yufei, could you make the argument for supporting a "secure" UDF? What
>>> use case are you addressing and what specifically changes about how the UDF
>>> is handled? If the idea is to hide the UDF definition, do we need to
>>> include it?
>>>
>>> I think this would be a signal to a "trusted engine". When the engine
>>> interacts with the catalog it sends authorization information about itself
>>> in addition to the user that it is acting on behalf of. That way the
>>> catalog knows that the secure UDF can be sent to the engine and won't be
>>> shown to the user. The majority of this logic is on the REST server side,
>>> and the only part that is communicated to the client is the request not to
>>> show the UDF to the user, right? In that case should this be a property
>>> rather than part of the definition? Even if we state that the client "must"
>>> suppress the UDF definition, it's really just a request. Only trusted
>>> engines can be passed the UDF definition, so a spec requirement to suppress
>>> the definition isn't very meaningful.
>>>
>>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Thanks for the summary, Ajantha!
>>>>
>>>> Multi-statement UDFs are definitely useful, but whether those
>>>> statements run within a single transaction should be treated as an
>>>> engine-level concern. The Iceberg UDF spec can spell out the expectation,
>>>> yet the actual guarantee still depends on the runtime. Even if a UDF
>>>> declares itself transactional, the engine may or may not enforce it.
>>>>
>>>> One more thing: should we also introduce a “secure UDF” option
>>>> supported by some engines[1], so the body and any sensitive details stay
>>>> hidden from callers?
>>>>
>>>> [1] https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <aj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks to everyone who joined the sync.
>>>>> Here is the meeting recording:
>>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>>> Summary:
>>>>>
>>>>>    - We have gone through the SQL UDF syntax supported by different
>>>>>    engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>>    - Each engine uses its own block separator, like $$ or '' or none.
>>>>>    Action item was to check whether engines support multi-statement
>>>>>    (transactional) UDF bodies.
>>>>>    - Discussed about function overloading. Need to check whether
>>>>>    these engines support function overloading for SQL UDFs. Postgres supports
>>>>>    it! If yes, need to adopt the spec to handle it.
>>>>>    - Started online spec review and discussed the deterministic flag
>>>>>    and concluded that we keep the independent fields (like deterministic) in
>>>>>    spec only if the majority of engines supports it. Else it will be passed in
>>>>>    a property bag (engine specific). And it is the engine's responsibility to
>>>>>    honor those optional properties.
>>>>>
>>>>> Feel free to review the current proposal document here
>>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>>
>>>>> Final spec will be put to review and vote once it is ready.
>>>>>
>>>>> Details for next Iceberg UDF sync:
>>>>>
>>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>> Google Meet joining info
>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>
>>>>> - Ajantha
>>>>>
>>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks to everyone who joined the sync.
>>>>>> Here is the meeting recording:
>>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>>
>>>>>> Summary:
>>>>>>
>>>>>>    -
>>>>>>
>>>>>>    We discussed including Python support; the majority agreed *not
>>>>>>    to* (see recording for details).
>>>>>>    -
>>>>>>
>>>>>>    No strong opposition to versioning — it will be included to
>>>>>>    support change tracking and similar use cases.
>>>>>>    -
>>>>>>
>>>>>>    Suggestions were made to document how each catalog resolves UDFs,
>>>>>>    similar to views and tables.
>>>>>>    -
>>>>>>
>>>>>>    We agreed not to deviate from the existing table/view spec —
>>>>>>    e.g., location will remain *required* for cross-catalog
>>>>>>    compatibility.
>>>>>>    -
>>>>>>
>>>>>>    We also discussed a bit about view interoperability as the same
>>>>>>    things are applicable here.
>>>>>>
>>>>>>    Feel free to review the proposal document
>>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>>    Final spec will be put to review and vote once it is ready.
>>>>>>
>>>>>> Details for next Iceberg UDF sync:
>>>>>>
>>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>>> Google Meet joining info
>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>
>>>>>> - Ajantha
>>>>>>
>>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF
>>>>>>> project. Everyone’s welcome to drop in and share ideas! Here is the meeting
>>>>>>> link:
>>>>>>>
>>>>>>> Iceberg UDF sync
>>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>>> Time zone: America/Los_Angeles
>>>>>>> Google Meet joining info
>>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>>
>>>>>>> Yufei
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Update on the progress.
>>>>>>>>
>>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>>>>>>> proposal. We covered several key points, though some are still open for
>>>>>>>> further discussion:
>>>>>>>>
>>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at this
>>>>>>>> stage? We explored the possibility of simplifying the specification by
>>>>>>>> avoiding view replication, and potentially introducing versioning support
>>>>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>>>>> versioning initially.
>>>>>>>>
>>>>>>>> b) *VarArgs Support*: While some query engines may not support
>>>>>>>> vararg syntax in CREATE FUNCTION, Iceberg UDFs could represent
>>>>>>>> such arguments as lists when supported by the engine.
>>>>>>>>
>>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support
>>>>>>>> generic types (e.g., object), we can only map engine-specific
>>>>>>>> types to Iceberg types. As a result, generic data types will not be
>>>>>>>> supported in the initial version.
>>>>>>>>
>>>>>>>> d) *Python Support*: Incorporating Python as a language for SQL
>>>>>>>> UDFs seems promising, especially given its potential to resolve
>>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>>> be captured in the specification.
>>>>>>>>
>>>>>>>> *Next Steps*
>>>>>>>> I will update the proposal document with two primary UDF use cases:
>>>>>>>>
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Policy exchange between engines
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    UDTF as a superset of view functionality
>>>>>>>>
>>>>>>>> The update will include corresponding syntax examples in both SQL
>>>>>>>> and Python, and detail how each use case is represented in Iceberg metadata.
>>>>>>>>
>>>>>>>> We also plan to set up regular syncs (open to more interested
>>>>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>>>>> - Ajantha
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <aj...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> I've updated the design document[1] based on the previous
>>>>>>>>> comments. Additionally, I've included the SQL UDF syntax supported by
>>>>>>>>> various vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>>
>>>>>>>>> I'm happy to schedule a separate sync if a deeper discussion is
>>>>>>>>> needed. Let's keep moving forward, especially with the renewed interest
>>>>>>>>> from the community.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>>
>>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hey everyone,
>>>>>>>>>>
>>>>>>>>>> During the last catalog community sync, there was significant
>>>>>>>>>> interest in storing UDFs in Iceberg and adding endpoints for UDF handling
>>>>>>>>>> in the REST catalog spec.
>>>>>>>>>>
>>>>>>>>>> I recently discussed this with Yufei to better understand the new
>>>>>>>>>> requirement of using UDFs for fine-grained access control policies. This
>>>>>>>>>> expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>>
>>>>>>>>>> Given the strong community interest and support, I’d like to take
>>>>>>>>>> ownership of this effort and revive the work. I'll be revisiting the
>>>>>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>>>>>> week.
>>>>>>>>>>
>>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>>> - Ajantha
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> The UDF spec does not require representations to be SQL. It
>>>>>>>>>>> merely does not specify (in this revision) how other representations are to
>>>>>>>>>>> be written.
>>>>>>>>>>>
>>>>>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>>>>>> "Representations" section).
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Dmitri.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It
>>>>>>>>>>>> leaves a way for future versions to add different representations later,
>>>>>>>>>>>> but only SQL is supported. That was also the feedback to my initial
>>>>>>>>>>>> skepticism about how it would work to add functions.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal
>>>>>>>>>>>>>> focuses on SQL-based engines, while Python-based systems often work with
>>>>>>>>>>>>>> data frames. Adding imperative languages like Python would make this
>>>>>>>>>>>>>> proposal more inclusive.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share among
>>>>>>>>>>>>>>> different engines."
>>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>>> I agree with this statement. I don't fully understand yet
>>>>>>>>>>>>>>> how the proposed design addresses shareability between the engines though.
>>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created functions
>>>>>>>>>>>>>>>> shareable
>>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in imperative
>>>>>>>>>>>>>>>> code?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg
>>>>>>>>>>>>>>>> UDFs are an interesting idea!
>>>>>>>>>>>>>>>> > Is there a plan to make the user-created functions
>>>>>>>>>>>>>>>> sharable between the engines?
>>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like in
>>>>>>>>>>>>>>>> e..g Spark or Trino?
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> I just looked through the proposal and added comments. I
>>>>>>>>>>>>>>>> think it would be helpful to also have a design doc that covers the choices
>>>>>>>>>>>>>>>> from the draft spec. For instance, the choice to enumerate all possible
>>>>>>>>>>>>>>>> function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> I think that the choice to enumerate function signatures
>>>>>>>>>>>>>>>> is limiting. It would be nice to see a discussion of the trade-offs and a
>>>>>>>>>>>>>>>> rationale for the choice. I think it would also be very helpful to have a
>>>>>>>>>>>>>>>> few representative use cases for this included in the doc. That way the
>>>>>>>>>>>>>>>> proposal can demonstrate that it solves those use cases with reasonable
>>>>>>>>>>>>>>>> trade-offs.
>>>>>>>>>>>>>>>> >> There are a few instances where this is inconsistent
>>>>>>>>>>>>>>>> with conventions in other specs. For example, using string IDs rather than
>>>>>>>>>>>>>>>> an integer.
>>>>>>>>>>>>>>>> >> This uses a very different model for spec versioning
>>>>>>>>>>>>>>>> than the Iceberg view and table specs. It requires readers to fail if there
>>>>>>>>>>>>>>>> are any unknown fields, which prevents the spec from adding things that are
>>>>>>>>>>>>>>>> fully backward-compatible. Other Iceberg specs only require a version
>>>>>>>>>>>>>>>> change to introduce forward-incompatible changes and I think that this
>>>>>>>>>>>>>>>> should do the same to avoid confusion.
>>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>>>>>>>>> version is associated with a single function signature.
>>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating functions
>>>>>>>>>>>>>>>> across engines, so this doesn’t show that the metadata proposed is
>>>>>>>>>>>>>>>> sufficient for cross-engine use cases.
>>>>>>>>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review comments,
>>>>>>>>>>>>>>>> I will raise a PR for spec addition next week.
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look at the
>>>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be
>>>>>>>>>>>>>>>> interested. Resharing since I do not think it was directly linked in the
>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any
>>>>>>>>>>>>>>>> review on the proposal.
>>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions
>>>>>>>>>>>>>>>> and how we want to implement it.
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin
>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of
>>>>>>>>>>>>>>>> (1) where the API is data flow/data pipeline API instead of SQL (e.g.,
>>>>>>>>>>>>>>>> Spark Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF
>>>>>>>>>>>>>>>> can be used to build a parameterized view.  So, there's definitely a lot in
>>>>>>>>>>>>>>>> common between UDFs and views.
>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin
>>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user
>>>>>>>>>>>>>>>> whose definition is a composition of other built-in functions/SQL
>>>>>>>>>>>>>>>> expressions.
>>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are
>>>>>>>>>>>>>>>> pretty much from (1) and I think those have more analogy to views due to
>>>>>>>>>>>>>>>> their SQL nature. Agree (2) is not practical to maintain by Iceberg, but I
>>>>>>>>>>>>>>>> think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common
>>>>>>>>>>>>>>>> set of functions across engines, I don't see how that is practical when
>>>>>>>>>>>>>>>> those engines are implemented so differently. Plugging in code -- and
>>>>>>>>>>>>>>>> especially custom user-supplied code -- seems inherently specialized to me
>>>>>>>>>>>>>>>> and should be part of the engines' design.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel
>>>>>>>>>>>>>>>> we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good
>>>>>>>>>>>>>>>> idea, even just a multi-dialect one like view, and that can allow engines
>>>>>>>>>>>>>>>> to for example parse a view SQL, and when a function referenced cannot be
>>>>>>>>>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the
>>>>>>>>>>>>>>>> actual proposal published.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable and
>>>>>>>>>>>>>>>> "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>>>>>>> views as well.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon
>>>>>>>>>>>>>>>> which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try to
>>>>>>>>>>>>>>>> tackle the idea to make views portable, which is conceptually not that much
>>>>>>>>>>>>>>>> different from portable UDFs.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative touch
>>>>>>>>>>>>>>>> to the idea of having UDFs in Iceberg, especially not in this early stage.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good idea
>>>>>>>>>>>>>>>> to add UDFs tracked by Iceberg catalogs. I think that Iceberg primarily
>>>>>>>>>>>>>>>> deals with things that are centralized, like tables of data. While it would
>>>>>>>>>>>>>>>> be great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat
>>>>>>>>>>>>>>>> <aj...@gmail.com> wrote:
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the community
>>>>>>>>>>>>>>>> interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for
>>>>>>>>>>>>>>>> storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views
>>>>>>>>>>>>>>>> in that they are associated with tables, but they can accept arguments and
>>>>>>>>>>>>>>>> produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>>>>>>> the translate layer).
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature
>>>>>>>>>>>>>>>> into Iceberg would be a valuable addition, and we're eager to collaborate
>>>>>>>>>>>>>>>> with the community to develop a UDF specification.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>> Databricks
>>>>>>>>>>>>
>>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Ryan Blue <rd...@gmail.com>]

Thanks for the extra detail. What do you think the spec would require?
Would it require hiding the UDF definition from users and require specific
pushdown cases be disabled? The use cases seem valid, but I'm trying to
understand the requirements this places on engines and why it needs to be
part of the spec, rather than part of the properties of the UDF.

On Fri, Jun 20, 2025 at 3:56 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi Ryan,
>
> Here are the main use cases for secure UDFs:
>
>    1.
>
>    Hiding UDF Definitions: This includes concealing the UDF body and
>    details like the list of imports, some of them aren’t applicable to SQL
>    UDFs.
>    2.
>
>    Sandboxed Execution: Ensuring the UDF runs in an isolated environment.
>    Again, this typically doesn’t apply to SQL UDFs.
>    3.
>
>    Preventing Data Leakage at Execution Time: For example, secure UDFs
>    may disable certain optimizations—such as predicate pushdown—to avoid
>    exposing sensitive data indirectly. [1]
>
> Given these scenarios, I agree with your point that the secure flag is
> primarily an instruction to the engine to behave differently. While it's
> largely an engine-side behavior, we still need to include this flag in the
> UDF definition to indicate whether a UDF is secure, especially considering
> the perf penalty introduced by scenario #3. We should clearly recommend
> that users avoid marking UDFs as secure unless it's truly necessary.
>
> [1]
> https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
> Yufei
>
>
> On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com> wrote:
>
>> Yufei, could you make the argument for supporting a "secure" UDF? What
>> use case are you addressing and what specifically changes about how the UDF
>> is handled? If the idea is to hide the UDF definition, do we need to
>> include it?
>>
>> I think this would be a signal to a "trusted engine". When the engine
>> interacts with the catalog it sends authorization information about itself
>> in addition to the user that it is acting on behalf of. That way the
>> catalog knows that the secure UDF can be sent to the engine and won't be
>> shown to the user. The majority of this logic is on the REST server side,
>> and the only part that is communicated to the client is the request not to
>> show the UDF to the user, right? In that case should this be a property
>> rather than part of the definition? Even if we state that the client "must"
>> suppress the UDF definition, it's really just a request. Only trusted
>> engines can be passed the UDF definition, so a spec requirement to suppress
>> the definition isn't very meaningful.
>>
>> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Thanks for the summary, Ajantha!
>>>
>>> Multi-statement UDFs are definitely useful, but whether those statements
>>> run within a single transaction should be treated as an engine-level
>>> concern. The Iceberg UDF spec can spell out the expectation, yet the actual
>>> guarantee still depends on the runtime. Even if a UDF declares itself
>>> transactional, the engine may or may not enforce it.
>>>
>>> One more thing: should we also introduce a “secure UDF” option supported
>>> by some engines[1], so the body and any sensitive details stay hidden from
>>> callers?
>>>
>>> [1] https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>>
>>> Yufei
>>>
>>>
>>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <aj...@gmail.com>
>>> wrote:
>>>
>>>> Thanks to everyone who joined the sync.
>>>> Here is the meeting recording:
>>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>>> Summary:
>>>>
>>>>    - We have gone through the SQL UDF syntax supported by different
>>>>    engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>>    - Each engine uses its own block separator, like $$ or '' or none.
>>>>    Action item was to check whether engines support multi-statement
>>>>    (transactional) UDF bodies.
>>>>    - Discussed about function overloading. Need to check whether these
>>>>    engines support function overloading for SQL UDFs. Postgres supports it! If
>>>>    yes, need to adopt the spec to handle it.
>>>>    - Started online spec review and discussed the deterministic flag
>>>>    and concluded that we keep the independent fields (like deterministic) in
>>>>    spec only if the majority of engines supports it. Else it will be passed in
>>>>    a property bag (engine specific). And it is the engine's responsibility to
>>>>    honor those optional properties.
>>>>
>>>> Feel free to review the current proposal document here
>>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>>
>>>> Final spec will be put to review and vote once it is ready.
>>>>
>>>> Details for next Iceberg UDF sync:
>>>>
>>>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>> Google Meet joining info
>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>
>>>> - Ajantha
>>>>
>>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks to everyone who joined the sync.
>>>>> Here is the meeting recording:
>>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>>
>>>>> Summary:
>>>>>
>>>>>    -
>>>>>
>>>>>    We discussed including Python support; the majority agreed *not to*
>>>>>    (see recording for details).
>>>>>    -
>>>>>
>>>>>    No strong opposition to versioning — it will be included to
>>>>>    support change tracking and similar use cases.
>>>>>    -
>>>>>
>>>>>    Suggestions were made to document how each catalog resolves UDFs,
>>>>>    similar to views and tables.
>>>>>    -
>>>>>
>>>>>    We agreed not to deviate from the existing table/view spec — e.g.,
>>>>>    location will remain *required* for cross-catalog compatibility.
>>>>>    -
>>>>>
>>>>>    We also discussed a bit about view interoperability as the same
>>>>>    things are applicable here.
>>>>>
>>>>>    Feel free to review the proposal document
>>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>>    With the current scope, it is similar to the view/table spec now.
>>>>>    Final spec will be put to review and vote once it is ready.
>>>>>
>>>>> Details for next Iceberg UDF sync:
>>>>>
>>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>>> Google Meet joining info
>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>
>>>>> - Ajantha
>>>>>
>>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF
>>>>>> project. Everyone’s welcome to drop in and share ideas! Here is the meeting
>>>>>> link:
>>>>>>
>>>>>> Iceberg UDF sync
>>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>>> Time zone: America/Los_Angeles
>>>>>> Google Meet joining info
>>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>>
>>>>>> Yufei
>>>>>>
>>>>>>
>>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <aj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Update on the progress.
>>>>>>>
>>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>>>>>> proposal. We covered several key points, though some are still open for
>>>>>>> further discussion:
>>>>>>>
>>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at this
>>>>>>> stage? We explored the possibility of simplifying the specification by
>>>>>>> avoiding view replication, and potentially introducing versioning support
>>>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>>>> versioning initially.
>>>>>>>
>>>>>>> b) *VarArgs Support*: While some query engines may not support
>>>>>>> vararg syntax in CREATE FUNCTION, Iceberg UDFs could represent such
>>>>>>> arguments as lists when supported by the engine.
>>>>>>>
>>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support
>>>>>>> generic types (e.g., object), we can only map engine-specific types
>>>>>>> to Iceberg types. As a result, generic data types will not be supported in
>>>>>>> the initial version.
>>>>>>>
>>>>>>> d) *Python Support*: Incorporating Python as a language for SQL
>>>>>>> UDFs seems promising, especially given its potential to resolve
>>>>>>> interoperability challenges. Some engines, however, require platform
>>>>>>> version and package dependency details to execute Python code—this should
>>>>>>> be captured in the specification.
>>>>>>>
>>>>>>> *Next Steps*
>>>>>>> I will update the proposal document with two primary UDF use cases:
>>>>>>>
>>>>>>>    -
>>>>>>>
>>>>>>>    Policy exchange between engines
>>>>>>>    -
>>>>>>>
>>>>>>>    UDTF as a superset of view functionality
>>>>>>>
>>>>>>> The update will include corresponding syntax examples in both SQL
>>>>>>> and Python, and detail how each use case is represented in Iceberg metadata.
>>>>>>>
>>>>>>> We also plan to set up regular syncs (open to more interested
>>>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>>>> - Ajantha
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> I've updated the design document[1] based on the previous comments.
>>>>>>>> Additionally, I've included the SQL UDF syntax supported by various
>>>>>>>> vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>>
>>>>>>>> I'm happy to schedule a separate sync if a deeper discussion is
>>>>>>>> needed. Let's keep moving forward, especially with the renewed interest
>>>>>>>> from the community.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>>
>>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <
>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hey everyone,
>>>>>>>>>
>>>>>>>>> During the last catalog community sync, there was significant
>>>>>>>>> interest in storing UDFs in Iceberg and adding endpoints for UDF handling
>>>>>>>>> in the REST catalog spec.
>>>>>>>>>
>>>>>>>>> I recently discussed this with Yufei to better understand the new
>>>>>>>>> requirement of using UDFs for fine-grained access control policies. This
>>>>>>>>> expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>>
>>>>>>>>> Given the strong community interest and support, I’d like to take
>>>>>>>>> ownership of this effort and revive the work. I'll be revisiting the
>>>>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>>>>> week.
>>>>>>>>>
>>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>>> - Ajantha
>>>>>>>>>
>>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> The UDF spec does not require representations to be SQL. It
>>>>>>>>>> merely does not specify (in this revision) how other representations are to
>>>>>>>>>> be written.
>>>>>>>>>>
>>>>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>>>>> "Representations" section).
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Dmitri.
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It leaves
>>>>>>>>>>> a way for future versions to add different representations later, but only
>>>>>>>>>>> SQL is supported. That was also the feedback to my initial skepticism about
>>>>>>>>>>> how it would work to add functions.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Dmitri.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal
>>>>>>>>>>>>> focuses on SQL-based engines, while Python-based systems often work with
>>>>>>>>>>>>> data frames. Adding imperative languages like Python would make this
>>>>>>>>>>>>> proposal more inclusive.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>> Fokko
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share among
>>>>>>>>>>>>>> different engines."
>>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>>> I agree with this statement. I don't fully understand yet how
>>>>>>>>>>>>>> the proposed design addresses shareability between the engines though.
>>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best
>>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Piotr, what do you mean by making user-created functions
>>>>>>>>>>>>>>> shareable
>>>>>>>>>>>>>>> between engines? Do you mean UDFs written in imperative code?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg
>>>>>>>>>>>>>>> UDFs are an interesting idea!
>>>>>>>>>>>>>>> > Is there a plan to make the user-created functions
>>>>>>>>>>>>>>> sharable between the engines?
>>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like in
>>>>>>>>>>>>>>> e..g Spark or Trino?
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> I just looked through the proposal and added comments. I
>>>>>>>>>>>>>>> think it would be helpful to also have a design doc that covers the choices
>>>>>>>>>>>>>>> from the draft spec. For instance, the choice to enumerate all possible
>>>>>>>>>>>>>>> function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> I think that the choice to enumerate function signatures
>>>>>>>>>>>>>>> is limiting. It would be nice to see a discussion of the trade-offs and a
>>>>>>>>>>>>>>> rationale for the choice. I think it would also be very helpful to have a
>>>>>>>>>>>>>>> few representative use cases for this included in the doc. That way the
>>>>>>>>>>>>>>> proposal can demonstrate that it solves those use cases with reasonable
>>>>>>>>>>>>>>> trade-offs.
>>>>>>>>>>>>>>> >> There are a few instances where this is inconsistent with
>>>>>>>>>>>>>>> conventions in other specs. For example, using string IDs rather than an
>>>>>>>>>>>>>>> integer.
>>>>>>>>>>>>>>> >> This uses a very different model for spec versioning than
>>>>>>>>>>>>>>> the Iceberg view and table specs. It requires readers to fail if there are
>>>>>>>>>>>>>>> any unknown fields, which prevents the spec from adding things that are
>>>>>>>>>>>>>>> fully backward-compatible. Other Iceberg specs only require a version
>>>>>>>>>>>>>>> change to introduce forward-incompatible changes and I think that this
>>>>>>>>>>>>>>> should do the same to avoid confusion.
>>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>>>>>>>> version is associated with a single function signature.
>>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating functions
>>>>>>>>>>>>>>> across engines, so this doesn’t show that the metadata proposed is
>>>>>>>>>>>>>>> sufficient for cross-engine use cases.
>>>>>>>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>>> >>> I will wait for a week and If no more review comments, I
>>>>>>>>>>>>>>> will raise a PR for spec addition next week.
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look at the
>>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be interested.
>>>>>>>>>>>>>>> Resharing since I do not think it was directly linked in the thread.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any
>>>>>>>>>>>>>>> review on the proposal.
>>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions and
>>>>>>>>>>>>>>> how we want to implement it.
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin Moustafa
>>>>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant
>>>>>>>>>>>>>>> scalar/aggregate/table user defined functions. Here are some examples of
>>>>>>>>>>>>>>> what I meant in (2):
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of (1)
>>>>>>>>>>>>>>> where the API is data flow/data pipeline API instead of SQL (e.g., Spark
>>>>>>>>>>>>>>> Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF
>>>>>>>>>>>>>>> can be used to build a parameterized view.  So, there's definitely a lot in
>>>>>>>>>>>>>>> common between UDFs and views.
>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin
>>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user whose
>>>>>>>>>>>>>>> definition is a composition of other built-in functions/SQL expressions.
>>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are
>>>>>>>>>>>>>>> pretty much from (1) and I think those have more analogy to views due to
>>>>>>>>>>>>>>> their SQL nature. Agree (2) is not practical to maintain by Iceberg, but I
>>>>>>>>>>>>>>> think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common set
>>>>>>>>>>>>>>> of functions across engines, I don't see how that is practical when those
>>>>>>>>>>>>>>> engines are implemented so differently. Plugging in code -- and especially
>>>>>>>>>>>>>>> custom user-supplied code -- seems inherently specialized to me and should
>>>>>>>>>>>>>>> be part of the engines' design.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel
>>>>>>>>>>>>>>> we can say exactly the same thing for Iceberg views, but yet we have
>>>>>>>>>>>>>>> Iceberg multi-dialect views implemented. Maybe it sounds like we are trying
>>>>>>>>>>>>>>> to draw a line between SQL vs other programming language as "code"? but I
>>>>>>>>>>>>>>> think SQL is just another type of code, and we are already talking about
>>>>>>>>>>>>>>> compiling all these different code dialects to an intermediate
>>>>>>>>>>>>>>> representation (using projects like Coral, Substrait), which will be stored
>>>>>>>>>>>>>>> as another type of representation of Iceberg view. I think the same
>>>>>>>>>>>>>>> functionality can be used for UDFs if developed.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good
>>>>>>>>>>>>>>> idea, even just a multi-dialect one like view, and that can allow engines
>>>>>>>>>>>>>>> to for example parse a view SQL, and when a function referenced cannot be
>>>>>>>>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the
>>>>>>>>>>>>>>> actual proposal published.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable and
>>>>>>>>>>>>>>> "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>>>>>> views as well.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon
>>>>>>>>>>>>>>> which engines can build, so the argument that UDFs aren't practical,
>>>>>>>>>>>>>>> because engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try to
>>>>>>>>>>>>>>> tackle the idea to make views portable, which is conceptually not that much
>>>>>>>>>>>>>>> different from portable UDFs.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative touch
>>>>>>>>>>>>>>> to the idea of having UDFs in Iceberg, especially not in this early stage.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good idea
>>>>>>>>>>>>>>> to add UDFs tracked by Iceberg catalogs. I think that Iceberg primarily
>>>>>>>>>>>>>>> deals with things that are centralized, like tables of data. While it would
>>>>>>>>>>>>>>> be great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the community
>>>>>>>>>>>>>>> interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for
>>>>>>>>>>>>>>> storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views in
>>>>>>>>>>>>>>> that they are associated with tables, but they can accept arguments and
>>>>>>>>>>>>>>> produce return values, or even function as inline expressions.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>>>>>> the translate layer).
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature
>>>>>>>>>>>>>>> into Iceberg would be a valuable addition, and we're eager to collaborate
>>>>>>>>>>>>>>> with the community to develop a UDF specification.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ryan Blue
>>>>>>>>>>> Databricks
>>>>>>>>>>>
>>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi Ryan,

Here are the main use cases for secure UDFs:

   1.

   Hiding UDF Definitions: This includes concealing the UDF body and
   details like the list of imports, some of them aren’t applicable to SQL
   UDFs.
   2.

   Sandboxed Execution: Ensuring the UDF runs in an isolated environment.
   Again, this typically doesn’t apply to SQL UDFs.
   3.

   Preventing Data Leakage at Execution Time: For example, secure UDFs may
   disable certain optimizations—such as predicate pushdown—to avoid exposing
   sensitive data indirectly. [1]

Given these scenarios, I agree with your point that the secure flag is
primarily an instruction to the engine to behave differently. While it's
largely an engine-side behavior, we still need to include this flag in the
UDF definition to indicate whether a UDF is secure, especially considering
the perf penalty introduced by scenario #3. We should clearly recommend
that users avoid marking UDFs as secure unless it's truly necessary.

[1]
https://docs.snowflake.com/en/developer-guide/pushdown-optimization#example-of-indirect-data-exposure-through-pushdown
Yufei


On Wed, Jun 18, 2025 at 12:32 PM Ryan Blue <rd...@gmail.com> wrote:

> Yufei, could you make the argument for supporting a "secure" UDF? What use
> case are you addressing and what specifically changes about how the UDF is
> handled? If the idea is to hide the UDF definition, do we need to include
> it?
>
> I think this would be a signal to a "trusted engine". When the engine
> interacts with the catalog it sends authorization information about itself
> in addition to the user that it is acting on behalf of. That way the
> catalog knows that the secure UDF can be sent to the engine and won't be
> shown to the user. The majority of this logic is on the REST server side,
> and the only part that is communicated to the client is the request not to
> show the UDF to the user, right? In that case should this be a property
> rather than part of the definition? Even if we state that the client "must"
> suppress the UDF definition, it's really just a request. Only trusted
> engines can be passed the UDF definition, so a spec requirement to suppress
> the definition isn't very meaningful.
>
> On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com> wrote:
>
>> Thanks for the summary, Ajantha!
>>
>> Multi-statement UDFs are definitely useful, but whether those statements
>> run within a single transaction should be treated as an engine-level
>> concern. The Iceberg UDF spec can spell out the expectation, yet the actual
>> guarantee still depends on the runtime. Even if a UDF declares itself
>> transactional, the engine may or may not enforce it.
>>
>> One more thing: should we also introduce a “secure UDF” option supported
>> by some engines[1], so the body and any sensitive details stay hidden from
>> callers?
>>
>> [1] https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>>
>> Yufei
>>
>>
>> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <aj...@gmail.com>
>> wrote:
>>
>>> Thanks to everyone who joined the sync.
>>> Here is the meeting recording:
>>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>>> Summary:
>>>
>>>    - We have gone through the SQL UDF syntax supported by different
>>>    engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>>    - Each engine uses its own block separator, like $$ or '' or none.
>>>    Action item was to check whether engines support multi-statement
>>>    (transactional) UDF bodies.
>>>    - Discussed about function overloading. Need to check whether these
>>>    engines support function overloading for SQL UDFs. Postgres supports it! If
>>>    yes, need to adopt the spec to handle it.
>>>    - Started online spec review and discussed the deterministic flag
>>>    and concluded that we keep the independent fields (like deterministic) in
>>>    spec only if the majority of engines supports it. Else it will be passed in
>>>    a property bag (engine specific). And it is the engine's responsibility to
>>>    honor those optional properties.
>>>
>>> Feel free to review the current proposal document here
>>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>>
>>> Final spec will be put to review and vote once it is ready.
>>>
>>> Details for next Iceberg UDF sync:
>>>
>>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>> Google Meet joining info
>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>
>>> - Ajantha
>>>
>>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com>
>>> wrote:
>>>
>>>> Thanks to everyone who joined the sync.
>>>> Here is the meeting recording:
>>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>>
>>>> Summary:
>>>>
>>>>    -
>>>>
>>>>    We discussed including Python support; the majority agreed *not to*
>>>>    (see recording for details).
>>>>    -
>>>>
>>>>    No strong opposition to versioning — it will be included to support
>>>>    change tracking and similar use cases.
>>>>    -
>>>>
>>>>    Suggestions were made to document how each catalog resolves UDFs,
>>>>    similar to views and tables.
>>>>    -
>>>>
>>>>    We agreed not to deviate from the existing table/view spec — e.g.,
>>>>    location will remain *required* for cross-catalog compatibility.
>>>>    -
>>>>
>>>>    We also discussed a bit about view interoperability as the same
>>>>    things are applicable here.
>>>>
>>>>    Feel free to review the proposal document
>>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>>    With the current scope, it is similar to the view/table spec now.
>>>>    Final spec will be put to review and vote once it is ready.
>>>>
>>>> Details for next Iceberg UDF sync:
>>>>
>>>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>>> Google Meet joining info
>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>
>>>> - Ajantha
>>>>
>>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> We’ve set up a dedicated bi-weekly community sync for the UDF project.
>>>>> Everyone’s welcome to drop in and share ideas! Here is the meeting link:
>>>>>
>>>>> Iceberg UDF sync
>>>>> Monday, June 2 · 9:00 – 10:00am
>>>>> Time zone: America/Los_Angeles
>>>>> Google Meet joining info
>>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>>
>>>>> Yufei
>>>>>
>>>>>
>>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <aj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Update on the progress.
>>>>>>
>>>>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>>>>> proposal. We covered several key points, though some are still open for
>>>>>> further discussion:
>>>>>>
>>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at this
>>>>>> stage? We explored the possibility of simplifying the specification by
>>>>>> avoiding view replication, and potentially introducing versioning support
>>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>>> versioning initially.
>>>>>>
>>>>>> b) *VarArgs Support*: While some query engines may not support
>>>>>> vararg syntax in CREATE FUNCTION, Iceberg UDFs could represent such
>>>>>> arguments as lists when supported by the engine.
>>>>>>
>>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support
>>>>>> generic types (e.g., object), we can only map engine-specific types
>>>>>> to Iceberg types. As a result, generic data types will not be supported in
>>>>>> the initial version.
>>>>>>
>>>>>> d) *Python Support*: Incorporating Python as a language for SQL UDFs
>>>>>> seems promising, especially given its potential to resolve interoperability
>>>>>> challenges. Some engines, however, require platform version and package
>>>>>> dependency details to execute Python code—this should be captured in the
>>>>>> specification.
>>>>>>
>>>>>> *Next Steps*
>>>>>> I will update the proposal document with two primary UDF use cases:
>>>>>>
>>>>>>    -
>>>>>>
>>>>>>    Policy exchange between engines
>>>>>>    -
>>>>>>
>>>>>>    UDTF as a superset of view functionality
>>>>>>
>>>>>> The update will include corresponding syntax examples in both SQL and
>>>>>> Python, and detail how each use case is represented in Iceberg metadata.
>>>>>>
>>>>>> We also plan to set up regular syncs (open to more interested
>>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>>> - Ajantha
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <aj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I've updated the design document[1] based on the previous comments.
>>>>>>> Additionally, I've included the SQL UDF syntax supported by various
>>>>>>> vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>>
>>>>>>> I'm happy to schedule a separate sync if a deeper discussion is
>>>>>>> needed. Let's keep moving forward, especially with the renewed interest
>>>>>>> from the community.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>>
>>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey everyone,
>>>>>>>>
>>>>>>>> During the last catalog community sync, there was significant
>>>>>>>> interest in storing UDFs in Iceberg and adding endpoints for UDF handling
>>>>>>>> in the REST catalog spec.
>>>>>>>>
>>>>>>>> I recently discussed this with Yufei to better understand the new
>>>>>>>> requirement of using UDFs for fine-grained access control policies. This
>>>>>>>> expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>>
>>>>>>>> Given the strong community interest and support, I’d like to take
>>>>>>>> ownership of this effort and revive the work. I'll be revisiting the
>>>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>>>> week.
>>>>>>>>
>>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>>> - Ajantha
>>>>>>>>
>>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> The UDF spec does not require representations to be SQL. It merely
>>>>>>>>> does not specify (in this revision) how other representations are to be
>>>>>>>>> written.
>>>>>>>>>
>>>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>>>> "Representations" section).
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Dmitri.
>>>>>>>>>
>>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It leaves
>>>>>>>>>> a way for future versions to add different representations later, but only
>>>>>>>>>> SQL is supported. That was also the feedback to my initial skepticism about
>>>>>>>>>> how it would work to add functions.
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Dmitri.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <
>>>>>>>>>>> fokko@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal focuses
>>>>>>>>>>>> on SQL-based engines, while Python-based systems often work with data
>>>>>>>>>>>> frames. Adding imperative languages like Python would make this proposal
>>>>>>>>>>>> more inclusive.
>>>>>>>>>>>>
>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>> Fokko
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share among
>>>>>>>>>>>>> different engines."
>>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>>> I agree with this statement. I don't fully understand yet how
>>>>>>>>>>>>> the proposed design addresses shareability between the engines though.
>>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best
>>>>>>>>>>>>> Piotr
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Piotr, what do you mean by making user-created functions
>>>>>>>>>>>>>> shareable
>>>>>>>>>>>>>> between engines? Do you mean UDFs written in imperative code?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg
>>>>>>>>>>>>>> UDFs are an interesting idea!
>>>>>>>>>>>>>> > Is there a plan to make the user-created functions sharable
>>>>>>>>>>>>>> between the engines?
>>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like in
>>>>>>>>>>>>>> e..g Spark or Trino?
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Best
>>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> I just looked through the proposal and added comments. I
>>>>>>>>>>>>>> think it would be helpful to also have a design doc that covers the choices
>>>>>>>>>>>>>> from the draft spec. For instance, the choice to enumerate all possible
>>>>>>>>>>>>>> function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> I think that the choice to enumerate function signatures
>>>>>>>>>>>>>> is limiting. It would be nice to see a discussion of the trade-offs and a
>>>>>>>>>>>>>> rationale for the choice. I think it would also be very helpful to have a
>>>>>>>>>>>>>> few representative use cases for this included in the doc. That way the
>>>>>>>>>>>>>> proposal can demonstrate that it solves those use cases with reasonable
>>>>>>>>>>>>>> trade-offs.
>>>>>>>>>>>>>> >> There are a few instances where this is inconsistent with
>>>>>>>>>>>>>> conventions in other specs. For example, using string IDs rather than an
>>>>>>>>>>>>>> integer.
>>>>>>>>>>>>>> >> This uses a very different model for spec versioning than
>>>>>>>>>>>>>> the Iceberg view and table specs. It requires readers to fail if there are
>>>>>>>>>>>>>> any unknown fields, which prevents the spec from adding things that are
>>>>>>>>>>>>>> fully backward-compatible. Other Iceberg specs only require a version
>>>>>>>>>>>>>> change to introduce forward-incompatible changes and I think that this
>>>>>>>>>>>>>> should do the same to avoid confusion.
>>>>>>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>>>>>>> version is associated with a single function signature.
>>>>>>>>>>>>>> >> There is no review of SQL syntax for creating functions
>>>>>>>>>>>>>> across engines, so this doesn’t show that the metadata proposed is
>>>>>>>>>>>>>> sufficient for cross-engine use cases.
>>>>>>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>>> >>> I will wait for a week and If no more review comments, I
>>>>>>>>>>>>>> will raise a PR for spec addition next week.
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> If anyone else is interested, please have a look at the
>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>>> >>>
>>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be interested.
>>>>>>>>>>>>>> Resharing since I do not think it was directly linked in the thread.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any
>>>>>>>>>>>>>> review on the proposal.
>>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions and
>>>>>>>>>>>>>> how we want to implement it.
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant scalar/aggregate/table
>>>>>>>>>>>>>> user defined functions. Here are some examples of what I meant in (2):
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of (1)
>>>>>>>>>>>>>> where the API is data flow/data pipeline API instead of SQL (e.g., Spark
>>>>>>>>>>>>>> Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF
>>>>>>>>>>>>>> can be used to build a parameterized view.  So, there's definitely a lot in
>>>>>>>>>>>>>> common between UDFs and views.
>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin
>>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user whose
>>>>>>>>>>>>>> definition is a composition of other built-in functions/SQL expressions.
>>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are
>>>>>>>>>>>>>> pretty much from (1) and I think those have more analogy to views due to
>>>>>>>>>>>>>> their SQL nature. Agree (2) is not practical to maintain by Iceberg, but I
>>>>>>>>>>>>>> think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common set
>>>>>>>>>>>>>> of functions across engines, I don't see how that is practical when those
>>>>>>>>>>>>>> engines are implemented so differently. Plugging in code -- and especially
>>>>>>>>>>>>>> custom user-supplied code -- seems inherently specialized to me and should
>>>>>>>>>>>>>> be part of the engines' design.
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel we
>>>>>>>>>>>>>> can say exactly the same thing for Iceberg views, but yet we have Iceberg
>>>>>>>>>>>>>> multi-dialect views implemented. Maybe it sounds like we are trying to draw
>>>>>>>>>>>>>> a line between SQL vs other programming language as "code"? but I think SQL
>>>>>>>>>>>>>> is just another type of code, and we are already talking about compiling
>>>>>>>>>>>>>> all these different code dialects to an intermediate representation (using
>>>>>>>>>>>>>> projects like Coral, Substrait), which will be stored as another type of
>>>>>>>>>>>>>> representation of Iceberg view. I think the same functionality can be used
>>>>>>>>>>>>>> for UDFs if developed.
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good
>>>>>>>>>>>>>> idea, even just a multi-dialect one like view, and that can allow engines
>>>>>>>>>>>>>> to for example parse a view SQL, and when a function referenced cannot be
>>>>>>>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the
>>>>>>>>>>>>>> actual proposal published.
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable and
>>>>>>>>>>>>>> "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>>>>> views as well.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon which
>>>>>>>>>>>>>> engines can build, so the argument that UDFs aren't practical, because
>>>>>>>>>>>>>> engines are different, is probably only a temporary concern.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try to
>>>>>>>>>>>>>> tackle the idea to make views portable, which is conceptually not that much
>>>>>>>>>>>>>> different from portable UDFs.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative touch
>>>>>>>>>>>>>> to the idea of having UDFs in Iceberg, especially not in this early stage.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good idea
>>>>>>>>>>>>>> to add UDFs tracked by Iceberg catalogs. I think that Iceberg primarily
>>>>>>>>>>>>>> deals with things that are centralized, like tables of data. While it would
>>>>>>>>>>>>>> be great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the community
>>>>>>>>>>>>>> interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for
>>>>>>>>>>>>>> storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views in
>>>>>>>>>>>>>> that they are associated with tables, but they can accept arguments and
>>>>>>>>>>>>>> produce return values, or even function as inline expressions.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>>>>> the translate layer).
>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature into
>>>>>>>>>>>>>> Iceberg would be a valuable addition, and we're eager to collaborate with
>>>>>>>>>>>>>> the community to develop a UDF specification.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> --
>>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ryan Blue
>>>>>>>>>> Databricks
>>>>>>>>>>
>>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Ryan Blue <rd...@gmail.com>]

Yufei, could you make the argument for supporting a "secure" UDF? What use
case are you addressing and what specifically changes about how the UDF is
handled? If the idea is to hide the UDF definition, do we need to include
it?

I think this would be a signal to a "trusted engine". When the engine
interacts with the catalog it sends authorization information about itself
in addition to the user that it is acting on behalf of. That way the
catalog knows that the secure UDF can be sent to the engine and won't be
shown to the user. The majority of this logic is on the REST server side,
and the only part that is communicated to the client is the request not to
show the UDF to the user, right? In that case should this be a property
rather than part of the definition? Even if we state that the client "must"
suppress the UDF definition, it's really just a request. Only trusted
engines can be passed the UDF definition, so a spec requirement to suppress
the definition isn't very meaningful.

On Mon, Jun 16, 2025 at 5:42 PM Yufei Gu <fl...@gmail.com> wrote:

> Thanks for the summary, Ajantha!
>
> Multi-statement UDFs are definitely useful, but whether those statements
> run within a single transaction should be treated as an engine-level
> concern. The Iceberg UDF spec can spell out the expectation, yet the actual
> guarantee still depends on the runtime. Even if a UDF declares itself
> transactional, the engine may or may not enforce it.
>
> One more thing: should we also introduce a “secure UDF” option supported
> by some engines[1], so the body and any sensitive details stay hidden from
> callers?
>
> [1] https://docs.snowflake.com/en/developer-guide/secure-udf-procedure
>
> Yufei
>
>
> On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
>> Thanks to everyone who joined the sync.
>> Here is the meeting recording:
>> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
>> Summary:
>>
>>    - We have gone through the SQL UDF syntax supported by different
>>    engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>>    - Each engine uses its own block separator, like $$ or '' or none.
>>    Action item was to check whether engines support multi-statement
>>    (transactional) UDF bodies.
>>    - Discussed about function overloading. Need to check whether these
>>    engines support function overloading for SQL UDFs. Postgres supports it! If
>>    yes, need to adopt the spec to handle it.
>>    - Started online spec review and discussed the deterministic flag and
>>    concluded that we keep the independent fields (like deterministic) in spec
>>    only if the majority of engines supports it. Else it will be passed in a
>>    property bag (engine specific). And it is the engine's responsibility to
>>    honor those optional properties.
>>
>> Feel free to review the current proposal document here
>> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>>
>> Final spec will be put to review and vote once it is ready.
>>
>> Details for next Iceberg UDF sync:
>>
>> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>> Google Meet joining info
>> Video call link: https://meet.google.com/aui-czix-nbh
>>
>> - Ajantha
>>
>> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com>
>> wrote:
>>
>>> Thanks to everyone who joined the sync.
>>> Here is the meeting recording:
>>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>>
>>> Summary:
>>>
>>>    -
>>>
>>>    We discussed including Python support; the majority agreed *not to*
>>>    (see recording for details).
>>>    -
>>>
>>>    No strong opposition to versioning — it will be included to support
>>>    change tracking and similar use cases.
>>>    -
>>>
>>>    Suggestions were made to document how each catalog resolves UDFs,
>>>    similar to views and tables.
>>>    -
>>>
>>>    We agreed not to deviate from the existing table/view spec — e.g.,
>>>    location will remain *required* for cross-catalog compatibility.
>>>    -
>>>
>>>    We also discussed a bit about view interoperability as the same
>>>    things are applicable here.
>>>
>>>    Feel free to review the proposal document
>>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>>    With the current scope, it is similar to the view/table spec now.
>>>    Final spec will be put to review and vote once it is ready.
>>>
>>> Details for next Iceberg UDF sync:
>>>
>>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>>> Google Meet joining info
>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>
>>> - Ajantha
>>>
>>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> We’ve set up a dedicated bi-weekly community sync for the UDF project.
>>>> Everyone’s welcome to drop in and share ideas! Here is the meeting link:
>>>>
>>>> Iceberg UDF sync
>>>> Monday, June 2 · 9:00 – 10:00am
>>>> Time zone: America/Los_Angeles
>>>> Google Meet joining info
>>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>>
>>>> Yufei
>>>>
>>>>
>>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <aj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Update on the progress.
>>>>>
>>>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>>>> proposal. We covered several key points, though some are still open for
>>>>> further discussion:
>>>>>
>>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at this
>>>>> stage? We explored the possibility of simplifying the specification by
>>>>> avoiding view replication, and potentially introducing versioning support
>>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>>> versioning initially.
>>>>>
>>>>> b) *VarArgs Support*: While some query engines may not support vararg
>>>>> syntax in CREATE FUNCTION, Iceberg UDFs could represent such
>>>>> arguments as lists when supported by the engine.
>>>>>
>>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support
>>>>> generic types (e.g., object), we can only map engine-specific types
>>>>> to Iceberg types. As a result, generic data types will not be supported in
>>>>> the initial version.
>>>>>
>>>>> d) *Python Support*: Incorporating Python as a language for SQL UDFs
>>>>> seems promising, especially given its potential to resolve interoperability
>>>>> challenges. Some engines, however, require platform version and package
>>>>> dependency details to execute Python code—this should be captured in the
>>>>> specification.
>>>>>
>>>>> *Next Steps*
>>>>> I will update the proposal document with two primary UDF use cases:
>>>>>
>>>>>    -
>>>>>
>>>>>    Policy exchange between engines
>>>>>    -
>>>>>
>>>>>    UDTF as a superset of view functionality
>>>>>
>>>>> The update will include corresponding syntax examples in both SQL and
>>>>> Python, and detail how each use case is represented in Iceberg metadata.
>>>>>
>>>>> We also plan to set up regular syncs (open to more interested
>>>>> participants) to continue refining and finalizing the UDF specification.
>>>>> - Ajantha
>>>>>
>>>>>
>>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <aj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I've updated the design document[1] based on the previous comments.
>>>>>> Additionally, I've included the SQL UDF syntax supported by various
>>>>>> vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>>
>>>>>> I'm happy to schedule a separate sync if a deeper discussion is
>>>>>> needed. Let's keep moving forward, especially with the renewed interest
>>>>>> from the community.
>>>>>>
>>>>>> [1]
>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>>
>>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <aj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey everyone,
>>>>>>>
>>>>>>> During the last catalog community sync, there was significant
>>>>>>> interest in storing UDFs in Iceberg and adding endpoints for UDF handling
>>>>>>> in the REST catalog spec.
>>>>>>>
>>>>>>> I recently discussed this with Yufei to better understand the new
>>>>>>> requirement of using UDFs for fine-grained access control policies. This
>>>>>>> expands the use cases beyond just versioned and interoperable UDFs.
>>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>>
>>>>>>> Given the strong community interest and support, I’d like to take
>>>>>>> ownership of this effort and revive the work. I'll be revisiting the
>>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>>> week.
>>>>>>>
>>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>>> - Ajantha
>>>>>>>
>>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>
>>>>>>>> The UDF spec does not require representations to be SQL. It merely
>>>>>>>> does not specify (in this revision) how other representations are to be
>>>>>>>> written.
>>>>>>>>
>>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>>> "Representations" section).
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Dmitri.
>>>>>>>>
>>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue
>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> Right now, SQL is an explicit requirement of the spec. It leaves a
>>>>>>>>> way for future versions to add different representations later, but only
>>>>>>>>> SQL is supported. That was also the feedback to my initial skepticism about
>>>>>>>>> how it would work to add functions.
>>>>>>>>>
>>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Dmitri.
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <fo...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal focuses
>>>>>>>>>>> on SQL-based engines, while Python-based systems often work with data
>>>>>>>>>>> frames. Adding imperative languages like Python would make this proposal
>>>>>>>>>>> more inclusive.
>>>>>>>>>>>
>>>>>>>>>>> Kind regards,
>>>>>>>>>>> Fokko
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>>>>>> "Without a common standard, the UDFs are hard to share among
>>>>>>>>>>>> different engines."
>>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>>> I agree with this statement. I don't fully understand yet how
>>>>>>>>>>>> the proposed design addresses shareability between the engines though.
>>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>>
>>>>>>>>>>>> Best
>>>>>>>>>>>> Piotr
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Piotr, what do you mean by making user-created functions
>>>>>>>>>>>>> shareable
>>>>>>>>>>>>> between engines? Do you mean UDFs written in imperative code?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Hi,
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg UDFs
>>>>>>>>>>>>> are an interesting idea!
>>>>>>>>>>>>> > Is there a plan to make the user-created functions sharable
>>>>>>>>>>>>> between the engines?
>>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like in
>>>>>>>>>>>>> e..g Spark or Trino?
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Best
>>>>>>>>>>>>> > Piotr
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> I just looked through the proposal and added comments. I
>>>>>>>>>>>>> think it would be helpful to also have a design doc that covers the choices
>>>>>>>>>>>>> from the draft spec. For instance, the choice to enumerate all possible
>>>>>>>>>>>>> function input struts rather than allowing generics and varargs.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> I think that the choice to enumerate function signatures is
>>>>>>>>>>>>> limiting. It would be nice to see a discussion of the trade-offs and a
>>>>>>>>>>>>> rationale for the choice. I think it would also be very helpful to have a
>>>>>>>>>>>>> few representative use cases for this included in the doc. That way the
>>>>>>>>>>>>> proposal can demonstrate that it solves those use cases with reasonable
>>>>>>>>>>>>> trade-offs.
>>>>>>>>>>>>> >> There are a few instances where this is inconsistent with
>>>>>>>>>>>>> conventions in other specs. For example, using string IDs rather than an
>>>>>>>>>>>>> integer.
>>>>>>>>>>>>> >> This uses a very different model for spec versioning than
>>>>>>>>>>>>> the Iceberg view and table specs. It requires readers to fail if there are
>>>>>>>>>>>>> any unknown fields, which prevents the spec from adding things that are
>>>>>>>>>>>>> fully backward-compatible. Other Iceberg specs only require a version
>>>>>>>>>>>>> change to introduce forward-incompatible changes and I think that this
>>>>>>>>>>>>> should do the same to avoid confusion.
>>>>>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>>>>>> version is associated with a single function signature.
>>>>>>>>>>>>> >> There is no review of SQL syntax for creating functions
>>>>>>>>>>>>> across engines, so this doesn’t show that the metadata proposed is
>>>>>>>>>>>>> sufficient for cross-engine use cases.
>>>>>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>>> >>> I will wait for a week and If no more review comments, I
>>>>>>>>>>>>> will raise a PR for spec addition next week.
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> If anyone else is interested, please have a look at the
>>>>>>>>>>>>> proposal
>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>>> >>>
>>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> I have left some comments. It is an interesting
>>>>>>>>>>>>> direction, but there might be some details that need to be fine tuned.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> The doc is here [1] for others who might be interested.
>>>>>>>>>>>>> Resharing since I do not think it was directly linked in the thread.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>>> >>>>
>>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any review
>>>>>>>>>>>>> on the proposal.
>>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>>> >>>>>
>>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions and
>>>>>>>>>>>>> how we want to implement it.
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant scalar/aggregate/table
>>>>>>>>>>>>> user defined functions. Here are some examples of what I meant in (2):
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of (1)
>>>>>>>>>>>>> where the API is data flow/data pipeline API instead of SQL (e.g., Spark
>>>>>>>>>>>>> Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF can
>>>>>>>>>>>>> be used to build a parameterized view.  So, there's definitely a lot in
>>>>>>>>>>>>> common between UDFs and views.
>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin
>>>>>>>>>>>>> Moustafa <wa...@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user whose
>>>>>>>>>>>>> definition is a composition of other built-in functions/SQL expressions.
>>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are
>>>>>>>>>>>>> pretty much from (1) and I think those have more analogy to views due to
>>>>>>>>>>>>> their SQL nature. Agree (2) is not practical to maintain by Iceberg, but I
>>>>>>>>>>>>> think Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common set
>>>>>>>>>>>>> of functions across engines, I don't see how that is practical when those
>>>>>>>>>>>>> engines are implemented so differently. Plugging in code -- and especially
>>>>>>>>>>>>> custom user-supplied code -- seems inherently specialized to me and should
>>>>>>>>>>>>> be part of the engines' design.
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel we
>>>>>>>>>>>>> can say exactly the same thing for Iceberg views, but yet we have Iceberg
>>>>>>>>>>>>> multi-dialect views implemented. Maybe it sounds like we are trying to draw
>>>>>>>>>>>>> a line between SQL vs other programming language as "code"? but I think SQL
>>>>>>>>>>>>> is just another type of code, and we are already talking about compiling
>>>>>>>>>>>>> all these different code dialects to an intermediate representation (using
>>>>>>>>>>>>> projects like Coral, Substrait), which will be stored as another type of
>>>>>>>>>>>>> representation of Iceberg view. I think the same functionality can be used
>>>>>>>>>>>>> for UDFs if developed.
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good
>>>>>>>>>>>>> idea, even just a multi-dialect one like view, and that can allow engines
>>>>>>>>>>>>> to for example parse a view SQL, and when a function referenced cannot be
>>>>>>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the
>>>>>>>>>>>>> actual proposal published.
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable and
>>>>>>>>>>>>> "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>>>> views as well.
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon which
>>>>>>>>>>>>> engines can build, so the argument that UDFs aren't practical, because
>>>>>>>>>>>>> engines are different, is probably only a temporary concern.
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try to
>>>>>>>>>>>>> tackle the idea to make views portable, which is conceptually not that much
>>>>>>>>>>>>> different from portable UDFs.
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative touch to
>>>>>>>>>>>>> the idea of having UDFs in Iceberg, especially not in this early stage.
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good idea to
>>>>>>>>>>>>> add UDFs tracked by Iceberg catalogs. I think that Iceberg primarily deals
>>>>>>>>>>>>> with things that are centralized, like tables of data. While it would be
>>>>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>>> penalty.
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the community
>>>>>>>>>>>>> interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for
>>>>>>>>>>>>> storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views in
>>>>>>>>>>>>> that they are associated with tables, but they can accept arguments and
>>>>>>>>>>>>> produce return values, or even function as inline expressions.
>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>>>> the translate layer).
>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature into
>>>>>>>>>>>>> Iceberg would be a valuable addition, and we're eager to collaborate with
>>>>>>>>>>>>> the community to develop a UDF specification.
>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> --
>>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ryan Blue
>>>>>>>>> Databricks
>>>>>>>>>
>>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Thanks for the summary, Ajantha!

Multi-statement UDFs are definitely useful, but whether those statements
run within a single transaction should be treated as an engine-level
concern. The Iceberg UDF spec can spell out the expectation, yet the actual
guarantee still depends on the runtime. Even if a UDF declares itself
transactional, the engine may or may not enforce it.

One more thing: should we also introduce a “secure UDF” option supported by
some engines[1], so the body and any sensitive details stay hidden from
callers?

[1] https://docs.snowflake.com/en/developer-guide/secure-udf-procedure

Yufei


On Mon, Jun 16, 2025 at 12:02 PM Ajantha Bhat <aj...@gmail.com> wrote:

> Thanks to everyone who joined the sync.
> Here is the meeting recording:
> https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
> Summary:
>
>    - We have gone through the SQL UDF syntax supported by different
>    engines (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
>    - Each engine uses its own block separator, like $$ or '' or none.
>    Action item was to check whether engines support multi-statement
>    (transactional) UDF bodies.
>    - Discussed about function overloading. Need to check whether these
>    engines support function overloading for SQL UDFs. Postgres supports it! If
>    yes, need to adopt the spec to handle it.
>    - Started online spec review and discussed the deterministic flag and
>    concluded that we keep the independent fields (like deterministic) in spec
>    only if the majority of engines supports it. Else it will be passed in a
>    property bag (engine specific). And it is the engine's responsibility to
>    honor those optional properties.
>
> Feel free to review the current proposal document here
> <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.
>
> Final spec will be put to review and vote once it is ready.
>
> Details for next Iceberg UDF sync:
>
> *Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
> Google Meet joining info
> Video call link: https://meet.google.com/aui-czix-nbh
>
> - Ajantha
>
> On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com> wrote:
>
>> Thanks to everyone who joined the sync.
>> Here is the meeting recording:
>> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>>
>> Summary:
>>
>>    -
>>
>>    We discussed including Python support; the majority agreed *not to*
>>    (see recording for details).
>>    -
>>
>>    No strong opposition to versioning — it will be included to support
>>    change tracking and similar use cases.
>>    -
>>
>>    Suggestions were made to document how each catalog resolves UDFs,
>>    similar to views and tables.
>>    -
>>
>>    We agreed not to deviate from the existing table/view spec — e.g.,
>>    location will remain *required* for cross-catalog compatibility.
>>    -
>>
>>    We also discussed a bit about view interoperability as the same
>>    things are applicable here.
>>
>>    Feel free to review the proposal document
>>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>>    With the current scope, it is similar to the view/table spec now.
>>    Final spec will be put to review and vote once it is ready.
>>
>> Details for next Iceberg UDF sync:
>>
>> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
>> Google Meet joining info
>> Video call link: https://meet.google.com/aui-czix-nbh
>>
>> - Ajantha
>>
>> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> We’ve set up a dedicated bi-weekly community sync for the UDF project.
>>> Everyone’s welcome to drop in and share ideas! Here is the meeting link:
>>>
>>> Iceberg UDF sync
>>> Monday, June 2 · 9:00 – 10:00am
>>> Time zone: America/Los_Angeles
>>> Google Meet joining info
>>> Video call link: https://meet.google.com/aui-czix-nbh
>>>
>>> Yufei
>>>
>>>
>>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <aj...@gmail.com>
>>> wrote:
>>>
>>>> Update on the progress.
>>>>
>>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>>> proposal. We covered several key points, though some are still open for
>>>> further discussion:
>>>>
>>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at this
>>>> stage? We explored the possibility of simplifying the specification by
>>>> avoiding view replication, and potentially introducing versioning support
>>>> later. UDTFs, being a superset of views in some ways, may not require
>>>> versioning initially.
>>>>
>>>> b) *VarArgs Support*: While some query engines may not support vararg
>>>> syntax in CREATE FUNCTION, Iceberg UDFs could represent such arguments
>>>> as lists when supported by the engine.
>>>>
>>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support generic
>>>> types (e.g., object), we can only map engine-specific types to Iceberg
>>>> types. As a result, generic data types will not be supported in the initial
>>>> version.
>>>>
>>>> d) *Python Support*: Incorporating Python as a language for SQL UDFs
>>>> seems promising, especially given its potential to resolve interoperability
>>>> challenges. Some engines, however, require platform version and package
>>>> dependency details to execute Python code—this should be captured in the
>>>> specification.
>>>>
>>>> *Next Steps*
>>>> I will update the proposal document with two primary UDF use cases:
>>>>
>>>>    -
>>>>
>>>>    Policy exchange between engines
>>>>    -
>>>>
>>>>    UDTF as a superset of view functionality
>>>>
>>>> The update will include corresponding syntax examples in both SQL and
>>>> Python, and detail how each use case is represented in Iceberg metadata.
>>>>
>>>> We also plan to set up regular syncs (open to more interested
>>>> participants) to continue refining and finalizing the UDF specification.
>>>> - Ajantha
>>>>
>>>>
>>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <aj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I've updated the design document[1] based on the previous comments.
>>>>> Additionally, I've included the SQL UDF syntax supported by various
>>>>> vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>>
>>>>> I'm happy to schedule a separate sync if a deeper discussion is
>>>>> needed. Let's keep moving forward, especially with the renewed interest
>>>>> from the community.
>>>>>
>>>>> [1]
>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>>
>>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <aj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hey everyone,
>>>>>>
>>>>>> During the last catalog community sync, there was significant
>>>>>> interest in storing UDFs in Iceberg and adding endpoints for UDF handling
>>>>>> in the REST catalog spec.
>>>>>>
>>>>>> I recently discussed this with Yufei to better understand the new
>>>>>> requirement of using UDFs for fine-grained access control policies. This
>>>>>> expands the use cases beyond just versioned and interoperable UDFs.
>>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>>
>>>>>> Given the strong community interest and support, I’d like to take
>>>>>> ownership of this effort and revive the work. I'll be revisiting the
>>>>>> document I proposed long back and will share an updated proposal by next
>>>>>> week.
>>>>>>
>>>>>> Looking forward to storing UDFs in Iceberg!
>>>>>> - Ajantha
>>>>>>
>>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>
>>>>>>> The UDF spec does not require representations to be SQL. It merely
>>>>>>> does not specify (in this revision) how other representations are to be
>>>>>>> written.
>>>>>>>
>>>>>>> This seems like an easy extension (adding a new type in the
>>>>>>> "Representations" section).
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Dmitri.
>>>>>>>
>>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue <bl...@databricks.com.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Right now, SQL is an explicit requirement of the spec. It leaves a
>>>>>>>> way for future versions to add different representations later, but only
>>>>>>>> SQL is supported. That was also the feedback to my initial skepticism about
>>>>>>>> how it would work to add functions.
>>>>>>>>
>>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>>
>>>>>>>>> I do not think the spec is meant to allow only SQL
>>>>>>>>> representations, although it is certainly faviouring SQL in examples... It
>>>>>>>>> would be nice to add a non-SQL example, indeed.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Dmitri.
>>>>>>>>>
>>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <fo...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal focuses
>>>>>>>>>> on SQL-based engines, while Python-based systems often work with data
>>>>>>>>>> frames. Adding imperative languages like Python would make this proposal
>>>>>>>>>> more inclusive.
>>>>>>>>>>
>>>>>>>>>> Kind regards,
>>>>>>>>>> Fokko
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>>>>> "Without a common standard, the UDFs are hard to share among
>>>>>>>>>>> different engines."
>>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>>> I agree with this statement. I don't fully understand yet how
>>>>>>>>>>> the proposed design addresses shareability between the engines though.
>>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>>
>>>>>>>>>>> Best
>>>>>>>>>>> Piotr
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>>
>>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Piotr, what do you mean by making user-created functions
>>>>>>>>>>>> shareable
>>>>>>>>>>>> between engines? Do you mean UDFs written in imperative code?
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>>> >
>>>>>>>>>>>> > Hi,
>>>>>>>>>>>> >
>>>>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg UDFs
>>>>>>>>>>>> are an interesting idea!
>>>>>>>>>>>> > Is there a plan to make the user-created functions sharable
>>>>>>>>>>>> between the engines?
>>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like in
>>>>>>>>>>>> e..g Spark or Trino?
>>>>>>>>>>>> >
>>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Best
>>>>>>>>>>>> > Piotr
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> I just looked through the proposal and added comments. I
>>>>>>>>>>>> think it would be helpful to also have a design doc that covers the choices
>>>>>>>>>>>> from the draft spec. For instance, the choice to enumerate all possible
>>>>>>>>>>>> function input struts rather than allowing generics and varargs.
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> I think that the choice to enumerate function signatures is
>>>>>>>>>>>> limiting. It would be nice to see a discussion of the trade-offs and a
>>>>>>>>>>>> rationale for the choice. I think it would also be very helpful to have a
>>>>>>>>>>>> few representative use cases for this included in the doc. That way the
>>>>>>>>>>>> proposal can demonstrate that it solves those use cases with reasonable
>>>>>>>>>>>> trade-offs.
>>>>>>>>>>>> >> There are a few instances where this is inconsistent with
>>>>>>>>>>>> conventions in other specs. For example, using string IDs rather than an
>>>>>>>>>>>> integer.
>>>>>>>>>>>> >> This uses a very different model for spec versioning than
>>>>>>>>>>>> the Iceberg view and table specs. It requires readers to fail if there are
>>>>>>>>>>>> any unknown fields, which prevents the spec from adding things that are
>>>>>>>>>>>> fully backward-compatible. Other Iceberg specs only require a version
>>>>>>>>>>>> change to introduce forward-incompatible changes and I think that this
>>>>>>>>>>>> should do the same to avoid confusion.
>>>>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>>>>> version is associated with a single function signature.
>>>>>>>>>>>> >> There is no review of SQL syntax for creating functions
>>>>>>>>>>>> across engines, so this doesn’t show that the metadata proposed is
>>>>>>>>>>>> sufficient for cross-engine use cases.
>>>>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>>> >>
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>>> >>> I will wait for a week and If no more review comments, I
>>>>>>>>>>>> will raise a PR for spec addition next week.
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>> If anyone else is interested, please have a look at the
>>>>>>>>>>>> proposal
>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>> I have left some comments. It is an interesting direction,
>>>>>>>>>>>> but there might be some details that need to be fine tuned.
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>> The doc is here [1] for others who might be interested.
>>>>>>>>>>>> Resharing since I do not think it was directly linked in the thread.
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>> [1]
>>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any review
>>>>>>>>>>>> on the proposal.
>>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions and
>>>>>>>>>>>> how we want to implement it.
>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>>> >>>>>>>
>>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant scalar/aggregate/table
>>>>>>>>>>>> user defined functions. Here are some examples of what I meant in (2):
>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of (1)
>>>>>>>>>>>> where the API is data flow/data pipeline API instead of SQL (e.g., Spark
>>>>>>>>>>>> Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF can
>>>>>>>>>>>> be used to build a parameterized view.  So, there's definitely a lot in
>>>>>>>>>>>> common between UDFs and views.
>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin Moustafa
>>>>>>>>>>>> <wa...@gmail.com> wrote:
>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user whose
>>>>>>>>>>>> definition is a composition of other built-in functions/SQL expressions.
>>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are pretty
>>>>>>>>>>>> much from (1) and I think those have more analogy to views due to their SQL
>>>>>>>>>>>> nature. Agree (2) is not practical to maintain by Iceberg, but I think
>>>>>>>>>>>> Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>> penalty.
>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec
>>>>>>>>>>>> (inspired by the view spec) this week to facilitate further discussions.
>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common set of
>>>>>>>>>>>> functions across engines, I don't see how that is practical when those
>>>>>>>>>>>> engines are implemented so differently. Plugging in code -- and especially
>>>>>>>>>>>> custom user-supplied code -- seems inherently specialized to me and should
>>>>>>>>>>>> be part of the engines' design.
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel we
>>>>>>>>>>>> can say exactly the same thing for Iceberg views, but yet we have Iceberg
>>>>>>>>>>>> multi-dialect views implemented. Maybe it sounds like we are trying to draw
>>>>>>>>>>>> a line between SQL vs other programming language as "code"? but I think SQL
>>>>>>>>>>>> is just another type of code, and we are already talking about compiling
>>>>>>>>>>>> all these different code dialects to an intermediate representation (using
>>>>>>>>>>>> projects like Coral, Substrait), which will be stored as another type of
>>>>>>>>>>>> representation of Iceberg view. I think the same functionality can be used
>>>>>>>>>>>> for UDFs if developed.
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good
>>>>>>>>>>>> idea, even just a multi-dialect one like view, and that can allow engines
>>>>>>>>>>>> to for example parse a view SQL, and when a function referenced cannot be
>>>>>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the
>>>>>>>>>>>> actual proposal published.
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable and
>>>>>>>>>>>> "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>>> views as well.
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon which
>>>>>>>>>>>> engines can build, so the argument that UDFs aren't practical, because
>>>>>>>>>>>> engines are different, is probably only a temporary concern.
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try to
>>>>>>>>>>>> tackle the idea to make views portable, which is conceptually not that much
>>>>>>>>>>>> different from portable UDFs.
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative touch to
>>>>>>>>>>>> the idea of having UDFs in Iceberg, especially not in this early stage.
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good idea to
>>>>>>>>>>>> add UDFs tracked by Iceberg catalogs. I think that Iceberg primarily deals
>>>>>>>>>>>> with things that are centralized, like tables of data. While it would be
>>>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>>> penalty.
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the community
>>>>>>>>>>>> interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for
>>>>>>>>>>>> storing the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views in
>>>>>>>>>>>> that they are associated with tables, but they can accept arguments and
>>>>>>>>>>>> produce return values, or even function as inline expressions.
>>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>>> the translate layer).
>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature into
>>>>>>>>>>>> Iceberg would be a valuable addition, and we're eager to collaborate with
>>>>>>>>>>>> the community to develop a UDF specification.
>>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>>> >>
>>>>>>>>>>>> >>
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> --
>>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>>> >> Databricks
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Databricks
>>>>>>>>
>>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Ajantha Bhat <aj...@gmail.com>]

Thanks to everyone who joined the sync.
Here is the meeting recording:
https://drive.google.com/file/d/10_Getaasv6tDMGzeZQUgcUVwCUAaFxiz/view?usp=sharing
Summary:

   - We have gone through the SQL UDF syntax supported by different engines
   (Snowflake, databricks, Dremio, Trino, OSS spark 4.0).
   - Each engine uses its own block separator, like $$ or '' or none.
   Action item was to check whether engines support multi-statement
   (transactional) UDF bodies.
   - Discussed about function overloading. Need to check whether these
   engines support function overloading for SQL UDFs. Postgres supports it! If
   yes, need to adopt the spec to handle it.
   - Started online spec review and discussed the deterministic flag and
   concluded that we keep the independent fields (like deterministic) in spec
   only if the majority of engines supports it. Else it will be passed in a
   property bag (engine specific). And it is the engine's responsibility to
   honor those optional properties.

Feel free to review the current proposal document here
<https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing>.

Final spec will be put to review and vote once it is ready.

Details for next Iceberg UDF sync:

*Monday, June 30 · 9:00 – 10:00am*Time zone: America/Los_Angeles
Google Meet joining info
Video call link: https://meet.google.com/aui-czix-nbh

- Ajantha

On Wed, Jun 4, 2025 at 9:00 PM Ajantha Bhat <aj...@gmail.com> wrote:

> Thanks to everyone who joined the sync.
> Here is the meeting recording:
> https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing
>
> Summary:
>
>    -
>
>    We discussed including Python support; the majority agreed *not to*
>    (see recording for details).
>    -
>
>    No strong opposition to versioning — it will be included to support
>    change tracking and similar use cases.
>    -
>
>    Suggestions were made to document how each catalog resolves UDFs,
>    similar to views and tables.
>    -
>
>    We agreed not to deviate from the existing table/view spec — e.g.,
>    location will remain *required* for cross-catalog compatibility.
>    -
>
>    We also discussed a bit about view interoperability as the same things
>    are applicable here.
>
>    Feel free to review the proposal document
>    <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0> here.
>    With the current scope, it is similar to the view/table spec now.
>    Final spec will be put to review and vote once it is ready.
>
> Details for next Iceberg UDF sync:
>
> *Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
> Google Meet joining info
> Video call link: https://meet.google.com/aui-czix-nbh
>
> - Ajantha
>
> On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com> wrote:
>
>> Hi folks,
>>
>> We’ve set up a dedicated bi-weekly community sync for the UDF project.
>> Everyone’s welcome to drop in and share ideas! Here is the meeting link:
>>
>> Iceberg UDF sync
>> Monday, June 2 · 9:00 – 10:00am
>> Time zone: America/Los_Angeles
>> Google Meet joining info
>> Video call link: https://meet.google.com/aui-czix-nbh
>>
>> Yufei
>>
>>
>> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <aj...@gmail.com>
>> wrote:
>>
>>> Update on the progress.
>>>
>>> I had a meeting today with Yufei and Yun.zou to discuss the UDF
>>> proposal. We covered several key points, though some are still open for
>>> further discussion:
>>>
>>> a) *UDF Versioning*: Do we truly need versioning for UDFs at this
>>> stage? We explored the possibility of simplifying the specification by
>>> avoiding view replication, and potentially introducing versioning support
>>> later. UDTFs, being a superset of views in some ways, may not require
>>> versioning initially.
>>>
>>> b) *VarArgs Support*: While some query engines may not support vararg
>>> syntax in CREATE FUNCTION, Iceberg UDFs could represent such arguments
>>> as lists when supported by the engine.
>>>
>>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support generic
>>> types (e.g., object), we can only map engine-specific types to Iceberg
>>> types. As a result, generic data types will not be supported in the initial
>>> version.
>>>
>>> d) *Python Support*: Incorporating Python as a language for SQL UDFs
>>> seems promising, especially given its potential to resolve interoperability
>>> challenges. Some engines, however, require platform version and package
>>> dependency details to execute Python code—this should be captured in the
>>> specification.
>>>
>>> *Next Steps*
>>> I will update the proposal document with two primary UDF use cases:
>>>
>>>    -
>>>
>>>    Policy exchange between engines
>>>    -
>>>
>>>    UDTF as a superset of view functionality
>>>
>>> The update will include corresponding syntax examples in both SQL and
>>> Python, and detail how each use case is represented in Iceberg metadata.
>>>
>>> We also plan to set up regular syncs (open to more interested
>>> participants) to continue refining and finalizing the UDF specification.
>>> - Ajantha
>>>
>>>
>>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <aj...@gmail.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I've updated the design document[1] based on the previous comments.
>>>> Additionally, I've included the SQL UDF syntax supported by various
>>>> vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>>
>>>> I'm happy to schedule a separate sync if a deeper discussion is needed.
>>>> Let's keep moving forward, especially with the renewed interest from the
>>>> community.
>>>>
>>>> [1]
>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>>
>>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <aj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> During the last catalog community sync, there was significant interest
>>>>> in storing UDFs in Iceberg and adding endpoints for UDF handling in the
>>>>> REST catalog spec.
>>>>>
>>>>> I recently discussed this with Yufei to better understand the new
>>>>> requirement of using UDFs for fine-grained access control policies. This
>>>>> expands the use cases beyond just versioned and interoperable UDFs.
>>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>>
>>>>> Given the strong community interest and support, I’d like to take
>>>>> ownership of this effort and revive the work. I'll be revisiting the
>>>>> document I proposed long back and will share an updated proposal by next
>>>>> week.
>>>>>
>>>>> Looking forward to storing UDFs in Iceberg!
>>>>> - Ajantha
>>>>>
>>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>
>>>>>> The UDF spec does not require representations to be SQL. It merely
>>>>>> does not specify (in this revision) how other representations are to be
>>>>>> written.
>>>>>>
>>>>>> This seems like an easy extension (adding a new type in the
>>>>>> "Representations" section).
>>>>>>
>>>>>> Cheers,
>>>>>> Dmitri.
>>>>>>
>>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue <bl...@databricks.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> Right now, SQL is an explicit requirement of the spec. It leaves a
>>>>>>> way for future versions to add different representations later, but only
>>>>>>> SQL is supported. That was also the feedback to my initial skepticism about
>>>>>>> how it would work to add functions.
>>>>>>>
>>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>>
>>>>>>>> I do not think the spec is meant to allow only SQL representations,
>>>>>>>> although it is certainly faviouring SQL in examples... It would be nice to
>>>>>>>> add a non-SQL example, indeed.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Dmitri.
>>>>>>>>
>>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <fo...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Coming from PyIceberg, I have concerns as this proposal focuses on
>>>>>>>>> SQL-based engines, while Python-based systems often work with data frames.
>>>>>>>>> Adding imperative languages like Python would make this proposal more
>>>>>>>>> inclusive.
>>>>>>>>>
>>>>>>>>> Kind regards,
>>>>>>>>> Fokko
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>>>> "Without a common standard, the UDFs are hard to share among
>>>>>>>>>> different engines."
>>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>>> I agree with this statement. I don't fully understand yet how the
>>>>>>>>>> proposed design addresses shareability between the engines though.
>>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>>
>>>>>>>>>> Best
>>>>>>>>>> Piotr
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>>
>>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Piotr, what do you mean by making user-created functions
>>>>>>>>>>> shareable
>>>>>>>>>>> between engines? Do you mean UDFs written in imperative code?
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > Hi,
>>>>>>>>>>> >
>>>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg UDFs
>>>>>>>>>>> are an interesting idea!
>>>>>>>>>>> > Is there a plan to make the user-created functions sharable
>>>>>>>>>>> between the engines?
>>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like in e..g
>>>>>>>>>>> Spark or Trino?
>>>>>>>>>>> >
>>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>>> >
>>>>>>>>>>> > Best
>>>>>>>>>>> > Piotr
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >> I just looked through the proposal and added comments. I
>>>>>>>>>>> think it would be helpful to also have a design doc that covers the choices
>>>>>>>>>>> from the draft spec. For instance, the choice to enumerate all possible
>>>>>>>>>>> function input struts rather than allowing generics and varargs.
>>>>>>>>>>> >>
>>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>>> >>
>>>>>>>>>>> >> I think that the choice to enumerate function signatures is
>>>>>>>>>>> limiting. It would be nice to see a discussion of the trade-offs and a
>>>>>>>>>>> rationale for the choice. I think it would also be very helpful to have a
>>>>>>>>>>> few representative use cases for this included in the doc. That way the
>>>>>>>>>>> proposal can demonstrate that it solves those use cases with reasonable
>>>>>>>>>>> trade-offs.
>>>>>>>>>>> >> There are a few instances where this is inconsistent with
>>>>>>>>>>> conventions in other specs. For example, using string IDs rather than an
>>>>>>>>>>> integer.
>>>>>>>>>>> >> This uses a very different model for spec versioning than the
>>>>>>>>>>> Iceberg view and table specs. It requires readers to fail if there are any
>>>>>>>>>>> unknown fields, which prevents the spec from adding things that are fully
>>>>>>>>>>> backward-compatible. Other Iceberg specs only require a version change to
>>>>>>>>>>> introduce forward-incompatible changes and I think that this should do the
>>>>>>>>>>> same to avoid confusion.
>>>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>>>> version is associated with a single function signature.
>>>>>>>>>>> >> There is no review of SQL syntax for creating functions
>>>>>>>>>>> across engines, so this doesn’t show that the metadata proposed is
>>>>>>>>>>> sufficient for cross-engine use cases.
>>>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>>> >>> I will wait for a week and If no more review comments, I
>>>>>>>>>>> will raise a PR for spec addition next week.
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> If anyone else is interested, please have a look at the
>>>>>>>>>>> proposal
>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> - Ajantha
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> I have left some comments. It is an interesting direction,
>>>>>>>>>>> but there might be some details that need to be fine tuned.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> The doc is here [1] for others who might be interested.
>>>>>>>>>>> Resharing since I do not think it was directly linked in the thread.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> [1]
>>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Thanks,
>>>>>>>>>>> >>>> Walaa.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any review
>>>>>>>>>>> on the proposal.
>>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>> >>>>>>>
>>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>>> >>>>>>>
>>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>>> >>>>>>>
>>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions and how
>>>>>>>>>>> we want to implement it.
>>>>>>>>>>> >>>>>>>
>>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>>> >>>>>>>
>>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant scalar/aggregate/table
>>>>>>>>>>> user defined functions. Here are some examples of what I meant in (2):
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of (1)
>>>>>>>>>>> where the API is data flow/data pipeline API instead of SQL (e.g., Spark
>>>>>>>>>>> Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>>
>>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>>> already be a huge step forward.
>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>> >>>>>>>>>
>>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF can
>>>>>>>>>>> be used to build a parameterized view.  So, there's definitely a lot in
>>>>>>>>>>> common between UDFs and views.
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is
>>>>>>>>>>> perceived as a "UDF". There are 2 flavors:
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user whose
>>>>>>>>>>> definition is a composition of other built-in functions/SQL expressions.
>>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are pretty
>>>>>>>>>>> much from (1) and I think those have more analogy to views due to their SQL
>>>>>>>>>>> nature. Agree (2) is not practical to maintain by Iceberg, but I think
>>>>>>>>>>> Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>> penalty.
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec (inspired
>>>>>>>>>>> by the view spec) this week to facilitate further discussions.
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common set of
>>>>>>>>>>> functions across engines, I don't see how that is practical when those
>>>>>>>>>>> engines are implemented so differently. Plugging in code -- and especially
>>>>>>>>>>> custom user-supplied code -- seems inherently specialized to me and should
>>>>>>>>>>> be part of the engines' design.
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel we
>>>>>>>>>>> can say exactly the same thing for Iceberg views, but yet we have Iceberg
>>>>>>>>>>> multi-dialect views implemented. Maybe it sounds like we are trying to draw
>>>>>>>>>>> a line between SQL vs other programming language as "code"? but I think SQL
>>>>>>>>>>> is just another type of code, and we are already talking about compiling
>>>>>>>>>>> all these different code dialects to an intermediate representation (using
>>>>>>>>>>> projects like Coral, Substrait), which will be stored as another type of
>>>>>>>>>>> representation of Iceberg view. I think the same functionality can be used
>>>>>>>>>>> for UDFs if developed.
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good idea,
>>>>>>>>>>> even just a multi-dialect one like view, and that can allow engines to for
>>>>>>>>>>> example parse a view SQL, and when a function referenced cannot be
>>>>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the
>>>>>>>>>>> actual proposal published.
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable and
>>>>>>>>>>> "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>>> views as well.
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon which
>>>>>>>>>>> engines can build, so the argument that UDFs aren't practical, because
>>>>>>>>>>> engines are different, is probably only a temporary concern.
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try to
>>>>>>>>>>> tackle the idea to make views portable, which is conceptually not that much
>>>>>>>>>>> different from portable UDFs.
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative touch to
>>>>>>>>>>> the idea of having UDFs in Iceberg, especially not in this early stage.
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good idea to
>>>>>>>>>>> add UDFs tracked by Iceberg catalogs. I think that Iceberg primarily deals
>>>>>>>>>>> with things that are centralized, like tables of data. While it would be
>>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>>> penalty.
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the community
>>>>>>>>>>> interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for storing
>>>>>>>>>>> the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views in
>>>>>>>>>>> that they are associated with tables, but they can accept arguments and
>>>>>>>>>>> produce return values, or even function as inline expressions.
>>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino,
>>>>>>>>>>> Snowflake, Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>>> the translate layer).
>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature into
>>>>>>>>>>> Iceberg would be a valuable addition, and we're eager to collaborate with
>>>>>>>>>>> the community to develop a UDF specification.
>>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>>> specification to propose to the community.
>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >>
>>>>>>>>>>> >> --
>>>>>>>>>>> >> Ryan Blue
>>>>>>>>>>> >> Databricks
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Databricks
>>>>>>>
>>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Ajantha Bhat <aj...@gmail.com>]

Thanks to everyone who joined the sync.
Here is the meeting recording:
https://drive.google.com/file/d/1WItItsNs3m3-no7_qWPHftGqVNOdpw5C/view?usp=sharing

Summary:

   -

   We discussed including Python support; the majority agreed *not to* (see
   recording for details).
   -

   No strong opposition to versioning — it will be included to support
   change tracking and similar use cases.
   -

   Suggestions were made to document how each catalog resolves UDFs,
   similar to views and tables.
   -

   We agreed not to deviate from the existing table/view spec — e.g.,
   location will remain *required* for cross-catalog compatibility.
   -

   We also discussed a bit about view interoperability as the same things
   are applicable here.

   Feel free to review the proposal document
   <https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?pli=1&tab=t.0>
here.
   With the current scope, it is similar to the view/table spec now.
   Final spec will be put to review and vote once it is ready.

Details for next Iceberg UDF sync:

*Monday, June 16 · 9:00 – 10:00am*Time zone: America/Los_Angeles
Google Meet joining info
Video call link: https://meet.google.com/aui-czix-nbh

- Ajantha

On Wed, May 21, 2025 at 3:33 AM Yufei Gu <fl...@gmail.com> wrote:

> Hi folks,
>
> We’ve set up a dedicated bi-weekly community sync for the UDF project.
> Everyone’s welcome to drop in and share ideas! Here is the meeting link:
>
> Iceberg UDF sync
> Monday, June 2 · 9:00 – 10:00am
> Time zone: America/Los_Angeles
> Google Meet joining info
> Video call link: https://meet.google.com/aui-czix-nbh
>
> Yufei
>
>
> On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
>> Update on the progress.
>>
>> I had a meeting today with Yufei and Yun.zou to discuss the UDF proposal.
>> We covered several key points, though some are still open for further
>> discussion:
>>
>> a) *UDF Versioning*: Do we truly need versioning for UDFs at this stage?
>> We explored the possibility of simplifying the specification by avoiding
>> view replication, and potentially introducing versioning support later.
>> UDTFs, being a superset of views in some ways, may not require versioning
>> initially.
>>
>> b) *VarArgs Support*: While some query engines may not support vararg
>> syntax in CREATE FUNCTION, Iceberg UDFs could represent such arguments
>> as lists when supported by the engine.
>>
>> c) *Generics in UDFs*: Since Iceberg currently doesn’t support generic
>> types (e.g., object), we can only map engine-specific types to Iceberg
>> types. As a result, generic data types will not be supported in the initial
>> version.
>>
>> d) *Python Support*: Incorporating Python as a language for SQL UDFs
>> seems promising, especially given its potential to resolve interoperability
>> challenges. Some engines, however, require platform version and package
>> dependency details to execute Python code—this should be captured in the
>> specification.
>>
>> *Next Steps*
>> I will update the proposal document with two primary UDF use cases:
>>
>>    -
>>
>>    Policy exchange between engines
>>    -
>>
>>    UDTF as a superset of view functionality
>>
>> The update will include corresponding syntax examples in both SQL and
>> Python, and detail how each use case is represented in Iceberg metadata.
>>
>> We also plan to set up regular syncs (open to more interested
>> participants) to continue refining and finalizing the UDF specification.
>> - Ajantha
>>
>>
>> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <aj...@gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I've updated the design document[1] based on the previous comments.
>>> Additionally, I've included the SQL UDF syntax supported by various
>>> vendors, including Dremio, Snowflake, Databricks, and Trino.
>>>
>>> I'm happy to schedule a separate sync if a deeper discussion is needed.
>>> Let's keep moving forward, especially with the renewed interest from the
>>> community.
>>>
>>> [1]
>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>>
>>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <aj...@gmail.com>
>>> wrote:
>>>
>>>> Hey everyone,
>>>>
>>>> During the last catalog community sync, there was significant interest
>>>> in storing UDFs in Iceberg and adding endpoints for UDF handling in the
>>>> REST catalog spec.
>>>>
>>>> I recently discussed this with Yufei to better understand the new
>>>> requirement of using UDFs for fine-grained access control policies. This
>>>> expands the use cases beyond just versioned and interoperable UDFs.
>>>> Additionally, I learnt that many vendors are interested in this feature.
>>>>
>>>> Given the strong community interest and support, I’d like to take
>>>> ownership of this effort and revive the work. I'll be revisiting the
>>>> document I proposed long back and will share an updated proposal by next
>>>> week.
>>>>
>>>> Looking forward to storing UDFs in Iceberg!
>>>> - Ajantha
>>>>
>>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>>> <dm...@dremio.com.invalid> wrote:
>>>>
>>>>> The UDF spec does not require representations to be SQL. It merely
>>>>> does not specify (in this revision) how other representations are to be
>>>>> written.
>>>>>
>>>>> This seems like an easy extension (adding a new type in the
>>>>> "Representations" section).
>>>>>
>>>>> Cheers,
>>>>> Dmitri.
>>>>>
>>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue <bl...@databricks.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> Right now, SQL is an explicit requirement of the spec. It leaves a
>>>>>> way for future versions to add different representations later, but only
>>>>>> SQL is supported. That was also the feedback to my initial skepticism about
>>>>>> how it would work to add functions.
>>>>>>
>>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>>
>>>>>>> I do not think the spec is meant to allow only SQL representations,
>>>>>>> although it is certainly faviouring SQL in examples... It would be nice to
>>>>>>> add a non-SQL example, indeed.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Dmitri.
>>>>>>>
>>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <fo...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Coming from PyIceberg, I have concerns as this proposal focuses on
>>>>>>>> SQL-based engines, while Python-based systems often work with data frames.
>>>>>>>> Adding imperative languages like Python would make this proposal more
>>>>>>>> inclusive.
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>> Fokko
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Walaa, thanks for asking!
>>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>>> "Without a common standard, the UDFs are hard to share among
>>>>>>>>> different engines."
>>>>>>>>> ("Background and Motivation" section).
>>>>>>>>> I agree with this statement. I don't fully understand yet how the
>>>>>>>>> proposed design addresses shareability between the engines though.
>>>>>>>>> I would use some help to understand this better.
>>>>>>>>>
>>>>>>>>> Best
>>>>>>>>> Piotr
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>>
>>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Piotr, what do you mean by making user-created functions shareable
>>>>>>>>>> between engines? Do you mean UDFs written in imperative code?
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>>> >
>>>>>>>>>> > Hi,
>>>>>>>>>> >
>>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg UDFs
>>>>>>>>>> are an interesting idea!
>>>>>>>>>> > Is there a plan to make the user-created functions sharable
>>>>>>>>>> between the engines?
>>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like in e..g
>>>>>>>>>> Spark or Trino?
>>>>>>>>>> >
>>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>>> >
>>>>>>>>>> > Best
>>>>>>>>>> > Piotr
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> I just looked through the proposal and added comments. I think
>>>>>>>>>> it would be helpful to also have a design doc that covers the choices from
>>>>>>>>>> the draft spec. For instance, the choice to enumerate all possible function
>>>>>>>>>> input struts rather than allowing generics and varargs.
>>>>>>>>>> >>
>>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>>> >>
>>>>>>>>>> >> I think that the choice to enumerate function signatures is
>>>>>>>>>> limiting. It would be nice to see a discussion of the trade-offs and a
>>>>>>>>>> rationale for the choice. I think it would also be very helpful to have a
>>>>>>>>>> few representative use cases for this included in the doc. That way the
>>>>>>>>>> proposal can demonstrate that it solves those use cases with reasonable
>>>>>>>>>> trade-offs.
>>>>>>>>>> >> There are a few instances where this is inconsistent with
>>>>>>>>>> conventions in other specs. For example, using string IDs rather than an
>>>>>>>>>> integer.
>>>>>>>>>> >> This uses a very different model for spec versioning than the
>>>>>>>>>> Iceberg view and table specs. It requires readers to fail if there are any
>>>>>>>>>> unknown fields, which prevents the spec from adding things that are fully
>>>>>>>>>> backward-compatible. Other Iceberg specs only require a version change to
>>>>>>>>>> introduce forward-incompatible changes and I think that this should do the
>>>>>>>>>> same to avoid confusion.
>>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>>> version is associated with a single function signature.
>>>>>>>>>> >> There is no review of SQL syntax for creating functions across
>>>>>>>>>> engines, so this doesn’t show that the metadata proposed is sufficient for
>>>>>>>>>> cross-engine use cases.
>>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>>> >>>
>>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>>> >>> I will wait for a week and If no more review comments, I will
>>>>>>>>>> raise a PR for spec addition next week.
>>>>>>>>>> >>>
>>>>>>>>>> >>> If anyone else is interested, please have a look at the
>>>>>>>>>> proposal
>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>> >>>
>>>>>>>>>> >>> - Ajantha
>>>>>>>>>> >>>
>>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> I have left some comments. It is an interesting direction,
>>>>>>>>>> but there might be some details that need to be fine tuned.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> The doc is here [1] for others who might be interested.
>>>>>>>>>> Resharing since I do not think it was directly linked in the thread.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> [1]
>>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Thanks,
>>>>>>>>>> >>>> Walaa.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any review on
>>>>>>>>>> the proposal.
>>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> - Ajantha
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions and how
>>>>>>>>>> we want to implement it.
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant scalar/aggregate/table
>>>>>>>>>> user defined functions. Here are some examples of what I meant in (2):
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of (1)
>>>>>>>>>> where the API is data flow/data pipeline API instead of SQL (e.g., Spark
>>>>>>>>>> Scala). Yes, that is also possible in the very long run :)
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>>> already be a huge step forward.
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF can be
>>>>>>>>>> used to build a parameterized view.  So, there's definitely a lot in common
>>>>>>>>>> between UDFs and views.
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>>> >>>>>>>>>>
>>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is perceived
>>>>>>>>>> as a "UDF". There are 2 flavors:
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user whose
>>>>>>>>>> definition is a composition of other built-in functions/SQL expressions.
>>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are pretty
>>>>>>>>>> much from (1) and I think those have more analogy to views due to their SQL
>>>>>>>>>> nature. Agree (2) is not practical to maintain by Iceberg, but I think
>>>>>>>>>> Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the proposal,
>>>>>>>>>> but I think this would be a very difficult area to tackle across engines,
>>>>>>>>>> languages, and memory models without having a huge performance penalty.
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec (inspired
>>>>>>>>>> by the view spec) this week to facilitate further discussions.
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common set of
>>>>>>>>>> functions across engines, I don't see how that is practical when those
>>>>>>>>>> engines are implemented so differently. Plugging in code -- and especially
>>>>>>>>>> custom user-supplied code -- seems inherently specialized to me and should
>>>>>>>>>> be part of the engines' design.
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel we can
>>>>>>>>>> say exactly the same thing for Iceberg views, but yet we have Iceberg
>>>>>>>>>> multi-dialect views implemented. Maybe it sounds like we are trying to draw
>>>>>>>>>> a line between SQL vs other programming language as "code"? but I think SQL
>>>>>>>>>> is just another type of code, and we are already talking about compiling
>>>>>>>>>> all these different code dialects to an intermediate representation (using
>>>>>>>>>> projects like Coral, Substrait), which will be stored as another type of
>>>>>>>>>> representation of Iceberg view. I think the same functionality can be used
>>>>>>>>>> for UDFs if developed.
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good idea,
>>>>>>>>>> even just a multi-dialect one like view, and that can allow engines to for
>>>>>>>>>> example parse a view SQL, and when a function referenced cannot be
>>>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the actual
>>>>>>>>>> proposal published.
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <
>>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable and
>>>>>>>>>> "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>>> views as well.
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon which
>>>>>>>>>> engines can build, so the argument that UDFs aren't practical, because
>>>>>>>>>> engines are different, is probably only a temporary concern.
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try to
>>>>>>>>>> tackle the idea to make views portable, which is conceptually not that much
>>>>>>>>>> different from portable UDFs.
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative touch to
>>>>>>>>>> the idea of having UDFs in Iceberg, especially not in this early stage.
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good idea to
>>>>>>>>>> add UDFs tracked by Iceberg catalogs. I think that Iceberg primarily deals
>>>>>>>>>> with things that are centralized, like tables of data. While it would be
>>>>>>>>>> great to have a common set of functions across engines, I don't see how
>>>>>>>>>> that is practical when those engines are implemented so differently.
>>>>>>>>>> Plugging in code -- and especially custom user-supplied code -- seems
>>>>>>>>>> inherently specialized to me and should be part of the engines' design.
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the
>>>>>>>>>> proposal, but I think this would be a very difficult area to tackle across
>>>>>>>>>> engines, languages, and memory models without having a huge performance
>>>>>>>>>> penalty.
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the community
>>>>>>>>>> interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for storing
>>>>>>>>>> the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views in that
>>>>>>>>>> they are associated with tables, but they can accept arguments and produce
>>>>>>>>>> return values, or even function as inline expressions.
>>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino, Snowflake,
>>>>>>>>>> Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>>> the translate layer).
>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature into
>>>>>>>>>> Iceberg would be a valuable addition, and we're eager to collaborate with
>>>>>>>>>> the community to develop a UDF specification.
>>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a
>>>>>>>>>> specification to propose to the community.
>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >> --
>>>>>>>>>> >> Ryan Blue
>>>>>>>>>> >> Databricks
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Databricks
>>>>>>
>>>>>

Re: [Discussion] Versioned SQL UDFs (Catalog routines) in Iceberg

[Yufei Gu <fl...@gmail.com>]

Hi folks,

We’ve set up a dedicated bi-weekly community sync for the UDF project.
Everyone’s welcome to drop in and share ideas! Here is the meeting link:

Iceberg UDF sync
Monday, June 2 · 9:00 – 10:00am
Time zone: America/Los_Angeles
Google Meet joining info
Video call link: https://meet.google.com/aui-czix-nbh

Yufei


On Fri, May 16, 2025 at 10:45 AM Ajantha Bhat <aj...@gmail.com> wrote:

> Update on the progress.
>
> I had a meeting today with Yufei and Yun.zou to discuss the UDF proposal.
> We covered several key points, though some are still open for further
> discussion:
>
> a) *UDF Versioning*: Do we truly need versioning for UDFs at this stage?
> We explored the possibility of simplifying the specification by avoiding
> view replication, and potentially introducing versioning support later.
> UDTFs, being a superset of views in some ways, may not require versioning
> initially.
>
> b) *VarArgs Support*: While some query engines may not support vararg
> syntax in CREATE FUNCTION, Iceberg UDFs could represent such arguments as
> lists when supported by the engine.
>
> c) *Generics in UDFs*: Since Iceberg currently doesn’t support generic
> types (e.g., object), we can only map engine-specific types to Iceberg
> types. As a result, generic data types will not be supported in the initial
> version.
>
> d) *Python Support*: Incorporating Python as a language for SQL UDFs
> seems promising, especially given its potential to resolve interoperability
> challenges. Some engines, however, require platform version and package
> dependency details to execute Python code—this should be captured in the
> specification.
>
> *Next Steps*
> I will update the proposal document with two primary UDF use cases:
>
>    -
>
>    Policy exchange between engines
>    -
>
>    UDTF as a superset of view functionality
>
> The update will include corresponding syntax examples in both SQL and
> Python, and detail how each use case is represented in Iceberg metadata.
>
> We also plan to set up regular syncs (open to more interested
> participants) to continue refining and finalizing the UDF specification.
> - Ajantha
>
>
> On Wed, Mar 12, 2025 at 9:16 PM Ajantha Bhat <aj...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I've updated the design document[1] based on the previous comments.
>> Additionally, I've included the SQL UDF syntax supported by various
>> vendors, including Dremio, Snowflake, Databricks, and Trino.
>>
>> I'm happy to schedule a separate sync if a deeper discussion is needed.
>> Let's keep moving forward, especially with the renewed interest from the
>> community.
>>
>> [1]
>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit?usp=sharing
>>
>> On Thu, Feb 13, 2025 at 11:17 PM Ajantha Bhat <aj...@gmail.com>
>> wrote:
>>
>>> Hey everyone,
>>>
>>> During the last catalog community sync, there was significant interest
>>> in storing UDFs in Iceberg and adding endpoints for UDF handling in the
>>> REST catalog spec.
>>>
>>> I recently discussed this with Yufei to better understand the new
>>> requirement of using UDFs for fine-grained access control policies. This
>>> expands the use cases beyond just versioned and interoperable UDFs.
>>> Additionally, I learnt that many vendors are interested in this feature.
>>>
>>> Given the strong community interest and support, I’d like to take
>>> ownership of this effort and revive the work. I'll be revisiting the
>>> document I proposed long back and will share an updated proposal by next
>>> week.
>>>
>>> Looking forward to storing UDFs in Iceberg!
>>> - Ajantha
>>>
>>> On Thu, Aug 8, 2024 at 2:55 PM Dmitri Bourlatchkov
>>> <dm...@dremio.com.invalid> wrote:
>>>
>>>> The UDF spec does not require representations to be SQL. It merely does
>>>> not specify (in this revision) how other representations are to be written.
>>>>
>>>> This seems like an easy extension (adding a new type in the
>>>> "Representations" section).
>>>>
>>>> Cheers,
>>>> Dmitri.
>>>>
>>>> On Thu, Aug 8, 2024 at 3:47 PM Ryan Blue <bl...@databricks.com.invalid>
>>>> wrote:
>>>>
>>>>> Right now, SQL is an explicit requirement of the spec. It leaves a way
>>>>> for future versions to add different representations later, but only SQL is
>>>>> supported. That was also the feedback to my initial skepticism about how it
>>>>> would work to add functions.
>>>>>
>>>>> On Thu, Aug 8, 2024 at 12:44 PM Dmitri Bourlatchkov
>>>>> <dm...@dremio.com.invalid> wrote:
>>>>>
>>>>>> I do not think the spec is meant to allow only SQL representations,
>>>>>> although it is certainly faviouring SQL in examples... It would be nice to
>>>>>> add a non-SQL example, indeed.
>>>>>>
>>>>>> Cheers,
>>>>>> Dmitri.
>>>>>>
>>>>>> On Thu, Aug 8, 2024 at 9:00 AM Fokko Driesprong <fo...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Coming from PyIceberg, I have concerns as this proposal focuses on
>>>>>>> SQL-based engines, while Python-based systems often work with data frames.
>>>>>>> Adding imperative languages like Python would make this proposal more
>>>>>>> inclusive.
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Fokko
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Op do 8 aug 2024 om 10:27 schreef Piotr Findeisen <
>>>>>>> piotr.findeisen@gmail.com>:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Walaa, thanks for asking!
>>>>>>>> In the design doc linked before  in this thread [1] i read
>>>>>>>> "Without a common standard, the UDFs are hard to share among
>>>>>>>> different engines."
>>>>>>>> ("Background and Motivation" section).
>>>>>>>> I agree with this statement. I don't fully understand yet how the
>>>>>>>> proposed design addresses shareability between the engines though.
>>>>>>>> I would use some help to understand this better.
>>>>>>>>
>>>>>>>> Best
>>>>>>>> Piotr
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [1] SQL User-Defined Function Spec
>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc
>>>>>>>>
>>>>>>>> On Wed, 7 Aug 2024 at 21:14, Walaa Eldin Moustafa <
>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Piotr, what do you mean by making user-created functions shareable
>>>>>>>>> between engines? Do you mean UDFs written in imperative code?
>>>>>>>>>
>>>>>>>>> On Wed, Aug 7, 2024 at 12:00 PM Piotr Findeisen
>>>>>>>>> <pi...@gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > Hi,
>>>>>>>>> >
>>>>>>>>> > Thank you Ajantha for creating this thread. The Iceberg UDFs are
>>>>>>>>> an interesting idea!
>>>>>>>>> > Is there a plan to make the user-created functions sharable
>>>>>>>>> between the engines?
>>>>>>>>> > If so, how would a CREATE FUNCTION statement look like in e..g
>>>>>>>>> Spark or Trino?
>>>>>>>>> >
>>>>>>>>> > Meanwhile, added a few comments in the doc.
>>>>>>>>> >
>>>>>>>>> > Best
>>>>>>>>> > Piotr
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On Thu, 1 Aug 2024 at 20:50, Ryan Blue
>>>>>>>>> <bl...@databricks.com.invalid> wrote:
>>>>>>>>> >>
>>>>>>>>> >> I just looked through the proposal and added comments. I think
>>>>>>>>> it would be helpful to also have a design doc that covers the choices from
>>>>>>>>> the draft spec. For instance, the choice to enumerate all possible function
>>>>>>>>> input struts rather than allowing generics and varargs.
>>>>>>>>> >>
>>>>>>>>> >> Here’s a quick summary of my feedback:
>>>>>>>>> >>
>>>>>>>>> >> I think that the choice to enumerate function signatures is
>>>>>>>>> limiting. It would be nice to see a discussion of the trade-offs and a
>>>>>>>>> rationale for the choice. I think it would also be very helpful to have a
>>>>>>>>> few representative use cases for this included in the doc. That way the
>>>>>>>>> proposal can demonstrate that it solves those use cases with reasonable
>>>>>>>>> trade-offs.
>>>>>>>>> >> There are a few instances where this is inconsistent with
>>>>>>>>> conventions in other specs. For example, using string IDs rather than an
>>>>>>>>> integer.
>>>>>>>>> >> This uses a very different model for spec versioning than the
>>>>>>>>> Iceberg view and table specs. It requires readers to fail if there are any
>>>>>>>>> unknown fields, which prevents the spec from adding things that are fully
>>>>>>>>> backward-compatible. Other Iceberg specs only require a version change to
>>>>>>>>> introduce forward-incompatible changes and I think that this should do the
>>>>>>>>> same to avoid confusion.
>>>>>>>>> >> It looks like the intent is to allow multiple function
>>>>>>>>> signatures per verison, but it is unclear how to encode them because a
>>>>>>>>> version is associated with a single function signature.
>>>>>>>>> >> There is no review of SQL syntax for creating functions across
>>>>>>>>> engines, so this doesn’t show that the metadata proposed is sufficient for
>>>>>>>>> cross-engine use cases.
>>>>>>>>> >> The example for a table-valued function shows a SELECT
>>>>>>>>> statement and it isn’t clear how this is distinct from a view
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> On Thu, Aug 1, 2024 at 3:15 AM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> Thanks Walaa and Robert for the review on this.
>>>>>>>>> >>>
>>>>>>>>> >>> We didn't find any blocker for the spec.
>>>>>>>>> >>> I will wait for a week and If no more review comments, I will
>>>>>>>>> raise a PR for spec addition next week.
>>>>>>>>> >>>
>>>>>>>>> >>> If anyone else is interested, please have a look at the
>>>>>>>>> proposal
>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>> >>>
>>>>>>>>> >>> - Ajantha
>>>>>>>>> >>>
>>>>>>>>> >>> On Tue, Jul 16, 2024 at 1:27 PM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> Hi Ajantha,
>>>>>>>>> >>>>
>>>>>>>>> >>>> I have left some comments. It is an interesting direction,
>>>>>>>>> but there might be some details that need to be fine tuned.
>>>>>>>>> >>>>
>>>>>>>>> >>>> The doc is here [1] for others who might be interested.
>>>>>>>>> Resharing since I do not think it was directly linked in the thread.
>>>>>>>>> >>>>
>>>>>>>>> >>>> [1]
>>>>>>>>> https://docs.google.com/document/d/1BDvOfhrH0ZQiQv9eLBqeAu8k8Vjfmeql9VzIiW1F0vc/edit
>>>>>>>>> >>>>
>>>>>>>>> >>>> Thanks,
>>>>>>>>> >>>> Walaa.
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Mon, Jul 15, 2024 at 11:09 PM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Hi, just another reminder since we didn't get any review on
>>>>>>>>> the proposal.
>>>>>>>>> >>>>> Initially proposed on June 4.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> - Ajantha
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Mon, Jun 24, 2024 at 4:21 PM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Hi everyone,
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> We've only received one review so far (from Benny).
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> We would appreciate more eyes on this.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> - Ajantha
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> On Tue, Jun 4, 2024 at 7:25 AM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Hi All,
>>>>>>>>> >>>>>>> Please find the proposal link
>>>>>>>>> >>>>>>> https://github.com/apache/iceberg/issues/10432
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Google doc link is attached in the proposal.
>>>>>>>>> >>>>>>> And Thanks Stephen Lin for working on it.
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Hope it gives more clarity to take the decisions and how
>>>>>>>>> we want to implement it.
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> - Ajantha
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> On Wed, May 29, 2024 at 4:01 AM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Thanks Jack. I actually meant scalar/aggregate/table user
>>>>>>>>> defined functions. Here are some examples of what I meant in (2):
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Hive GenericUDF:
>>>>>>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
>>>>>>>>> >>>>>>>> Trino user defined functions:
>>>>>>>>> https://trino.io/docs/current/develop/functions.html
>>>>>>>>> >>>>>>>> Flink user defined functions:
>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/functions/udfs/
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Probably what you referred to is a variation of (1) where
>>>>>>>>> the API is data flow/data pipeline API instead of SQL (e.g., Spark Scala).
>>>>>>>>> Yes, that is also possible in the very long run :)
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Thanks,
>>>>>>>>> >>>>>>>> Walaa.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> On Tue, May 28, 2024 at 2:57 PM Jack Ye <
>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> > (2) Custom code written in imperative function
>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> I think we could still explore some long term
>>>>>>>>> opportunities in this case. Consider you register a Spark temp view as some
>>>>>>>>> sort of data frame read, then it could still be resolved to a Spark plan
>>>>>>>>> that is representable by an intermediate representation. But I agree this
>>>>>>>>> gets very complicated very soon, and just having the case (1) covered would
>>>>>>>>> already be a huge step forward.
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> -Jack
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> On Tue, May 28, 2024 at 1:40 PM Benny Chow <
>>>>>>>>> btchow@gmail.com> wrote:
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> It's interesting to note that a tabular SQL UDF can be
>>>>>>>>> used to build a parameterized view.  So, there's definitely a lot in common
>>>>>>>>> between UDFs and views.
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> Thanks
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2024 at 9:53 AM Walaa Eldin Moustafa <
>>>>>>>>> wa.moustafa@gmail.com> wrote:
>>>>>>>>> >>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>> I think there is a disconnect about what is perceived
>>>>>>>>> as a "UDF". There are 2 flavors:
>>>>>>>>> >>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>> (1) Functions that are defined by the user whose
>>>>>>>>> definition is a composition of other built-in functions/SQL expressions.
>>>>>>>>> >>>>>>>>>>> (2) Custom code written in imperative function
>>>>>>>>> according to a Java/Scala/Python API, etc.
>>>>>>>>> >>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>> All the examples in Ajantha's references are pretty
>>>>>>>>> much from (1) and I think those have more analogy to views due to their SQL
>>>>>>>>> nature. Agree (2) is not practical to maintain by Iceberg, but I think
>>>>>>>>> Ajantha's use cases are around (1), and may be worth evaluating.
>>>>>>>>> >>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>> Thanks,
>>>>>>>>> >>>>>>>>>>> Walaa.
>>>>>>>>> >>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2024 at 9:45 AM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> I guess we'll know more when you post the proposal,
>>>>>>>>> but I think this would be a very difficult area to tackle across engines,
>>>>>>>>> languages, and memory models without having a huge performance penalty.
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> Assuming Iceberg initially supports SQL
>>>>>>>>> representations of UDFs (similar to views as shared by the reference links
>>>>>>>>> above), the complexity involved will be similar to managing views.
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> Thanks, Ryan, Robert, and Jack, for your input.
>>>>>>>>> >>>>>>>>>>>> We will work on publishing the draft spec (inspired
>>>>>>>>> by the view spec) this week to facilitate further discussions.
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> - Ajantha
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2024 at 7:33 PM Jack Ye <
>>>>>>>>> yezhaoqin@gmail.com> wrote:
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> > While it would be great to have a common set of
>>>>>>>>> functions across engines, I don't see how that is practical when those
>>>>>>>>> engines are implemented so differently. Plugging in code -- and especially
>>>>>>>>> custom user-supplied code -- seems inherently specialized to me and should
>>>>>>>>> be part of the engines' design.
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> How is this different from the views? I feel we can
>>>>>>>>> say exactly the same thing for Iceberg views, but yet we have Iceberg
>>>>>>>>> multi-dialect views implemented. Maybe it sounds like we are trying to draw
>>>>>>>>> a line between SQL vs other programming language as "code"? but I think SQL
>>>>>>>>> is just another type of code, and we are already talking about compiling
>>>>>>>>> all these different code dialects to an intermediate representation (using
>>>>>>>>> projects like Coral, Substrait), which will be stored as another type of
>>>>>>>>> representation of Iceberg view. I think the same functionality can be used
>>>>>>>>> for UDFs if developed.
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> I actually hink adding UDF support is a good idea,
>>>>>>>>> even just a multi-dialect one like view, and that can allow engines to for
>>>>>>>>> example parse a view SQL, and when a function referenced cannot be
>>>>>>>>> resolved, try to seek for a multi-dialect UDF definition.
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> I guess we can discuss more when we have the actual
>>>>>>>>> proposal published.
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> Best,
>>>>>>>>> >>>>>>>>>>>>> Jack Ye
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> On Tue, May 28, 2024 at 1:32 AM Robert Stupp <
>>>>>>>>> snazy@snazy.de> wrote:
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> UDFs are as engine specific and portable and
>>>>>>>>> "non-centralized" as views are. The same performance concerns apply to
>>>>>>>>> views as well.
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> Iceberg should define a common base upon which
>>>>>>>>> engines can build, so the argument that UDFs aren't practical, because
>>>>>>>>> engines are different, is probably only a temporary concern.
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> In the long term, Iceberg should also try to tackle
>>>>>>>>> the idea to make views portable, which is conceptually not that much
>>>>>>>>> different from portable UDFs.
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> PS: I'm not a fan of adding a negative touch to the
>>>>>>>>> idea of having UDFs in Iceberg, especially not in this early stage.
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> On 24.05.24 20:53, Ryan Blue wrote:
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> Thanks, Ajantha.
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> I'm skeptical about whether it's a good idea to add
>>>>>>>>> UDFs tracked by Iceberg catalogs. I think that Iceberg primarily deals with
>>>>>>>>> things that are centralized, like tables of data. While it would be great
>>>>>>>>> to have a common set of functions across engines, I don't see how that is
>>>>>>>>> practical when those engines are implemented so differently. Plugging in
>>>>>>>>> code -- and especially custom user-supplied code -- seems inherently
>>>>>>>>> specialized to me and should be part of the engines' design.
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> I guess we'll know more when you post the proposal,
>>>>>>>>> but I think this would be a very difficult area to tackle across engines,
>>>>>>>>> languages, and memory models without having a huge performance penalty.
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> Ryan
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 24, 2024 at 8:10 AM Ajantha Bhat <
>>>>>>>>> ajanthabhat@gmail.com> wrote:
>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>> This is a discussion to gauge the community
>>>>>>>>> interest in storing the Versioned SQL UDFs in Iceberg.
>>>>>>>>> >>>>>>>>>>>>>>> We want to propose the spec addition for storing
>>>>>>>>> the versioned UDFs in Iceberg (inspired by view spec).
>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>> These UDFs can operate similarly to views in that
>>>>>>>>> they are associated with tables, but they can accept arguments and produce
>>>>>>>>> return values, or even function as inline expressions.
>>>>>>>>> >>>>>>>>>>>>>>> Many Query engines like Dremio, Trino, Snowflake,
>>>>>>>>> Databricks Spark supports SQL UDFs at catalog level [1].
>>>>>>>>> >>>>>>>>>>>>>>> But storing them in Iceberg can enable
>>>>>>>>> >>>>>>>>>>>>>>> - Versioning of these UDFs.
>>>>>>>>> >>>>>>>>>>>>>>> - Interoperability between the engines.
>>>>>>>>> Potentially engines can understand the UDFs written by other engines (with
>>>>>>>>> the translate layer).
>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>> We believe that integrating this feature into
>>>>>>>>> Iceberg would be a valuable addition, and we're eager to collaborate with
>>>>>>>>> the community to develop a UDF specification.
>>>>>>>>> >>>>>>>>>>>>>>> Stephen has already begun drafting a specification
>>>>>>>>> to propose to the community.
>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>> Let us know your thoughts on this.
>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>> [1]
>>>>>>>>> >>>>>>>>>>>>>>> Dremio -
>>>>>>>>> https://docs.dremio.com/current/reference/sql/commands/functions#creating-a-function
>>>>>>>>> >>>>>>>>>>>>>>> Trino -
>>>>>>>>> https://trino.io/docs/current/sql/create-function.html
>>>>>>>>> >>>>>>>>>>>>>>> Snowflake -
>>>>>>>>> https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions
>>>>>>>>> >>>>>>>>>>>>>>> Databricks -
>>>>>>>>> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-sql-function.html
>>>>>>>>> >>>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>> - Ajantha
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>> >>>>>>>>>>>>>> Ryan Blue
>>>>>>>>> >>>>>>>>>>>>>> Tabular
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> --
>>>>>>>>> >>>>>>>>>>>>>> Robert Stupp
>>>>>>>>> >>>>>>>>>>>>>> @snazy
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> --
>>>>>>>>> >> Ryan Blue
>>>>>>>>> >> Databricks
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Databricks
>>>>>
>>>>