You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@paimon.apache.org by lining jing <ji...@gmail.com> on 2025/04/27 03:38:55 UTC
Support Function
Hi devs,
I would like to start a discussion about PIP-31: Introduce Function
As Paimon has supported table and view， maybe we could support function，
and we could use it in：
1. Unified column-level filtering and processing capabilities (ps: for
email information encryption and decryption)
2. Parameterized view capabilities: similar to retrieving views, it
supports parameters
Looking forward to your feedback, thanks!
[1] pip-31
<https://cwiki.apache.org/confluence/display/PAIMON/PIP-31%3A+Introduce+Function>
Best, Jerry
Re: Support Function
Posted by yu zelin <yu...@gmail.com>.
Thanks Jerry for starting this discussion. +1 for it.
Best,
Zelin Yu
On Sun, Apr 27, 2025 at 11:48 AM lining jing <ji...@gmail.com> wrote:
> Hi devs,
> I would like to start a discussion about PIP-31: Introduce Function
>
> As Paimon has supported table and view， maybe we could support function，
> and we could use it in：
> 1. Unified column-level filtering and processing capabilities (ps: for
> email information encryption and decryption)
> 2. Parameterized view capabilities: similar to retrieving views, it
> supports parameters
>
> Looking forward to your feedback, thanks!
> [1] pip-31
> <
> https://cwiki.apache.org/confluence/display/PAIMON/PIP-31%3A+Introduce+Function
> >
>
> Best, Jerry
>
Re: Support Function
Posted by JUNHAO YE <ye...@gmail.com>.
+1 For me, thanks for explanation.
> 2025年4月28日 10:23，lining jing <ji...@gmail.com> 写道：
>
> For function, SQL function don't suffer from compatibility problems due to
> language issues and depends only on the engine. Therefore, SQL function is
> recommended, but not all engines support SQL function. Trino supports SQL
> function and Python function, but since SQL function can override Python
> function's capabilities, only keep SQL function.
>
> Best, Jerry
>
> yejunhao <44...@qq.com.invalid> 于2025年4月28日周一 09:59写道：
>
>> Looks nice,
>> Is that means, we can define our UDF in paimon? And could you provide some
>> Interface of the java?
>> And why trino only support sql function not jar function?
>>
>> Best,
>> Junhao
>>
>>> 2025年4月27日 11:38，lining jing <ji...@gmail.com> 写道：
>>>
>>> Hi devs,
>>> I would like to start a discussion about PIP-31: Introduce Function
>>>
>>> As Paimon has supported table and view， maybe we could support function，
>>> and we could use it in：
>>> 1. Unified column-level filtering and processing capabilities (ps: for
>>> email information encryption and decryption)
>>> 2. Parameterized view capabilities: similar to retrieving views, it
>>> supports parameters
>>>
>>> Looking forward to your feedback, thanks!
>>> [1] pip-31
>>> <
>> https://cwiki.apache.org/confluence/display/PAIMON/PIP-31%3A+Introduce+Function
>>>
>>>
>>> Best, Jerry
>>
>>
Re: Support Function
Posted by lining jing <ji...@gmail.com>.
For function, SQL function don't suffer from compatibility problems due to
language issues and depends only on the engine. Therefore, SQL function is
recommended, but not all engines support SQL function. Trino supports SQL
function and Python function, but since SQL function can override Python
function's capabilities, only keep SQL function.
Best, Jerry
yejunhao <44...@qq.com.invalid> 于2025年4月28日周一 09:59写道：
> Looks nice,
> Is that means, we can define our UDF in paimon? And could you provide some
> Interface of the java?
> And why trino only support sql function not jar function?
>
> Best,
> Junhao
>
> > 2025年4月27日 11:38，lining jing <ji...@gmail.com> 写道：
> >
> > Hi devs,
> > I would like to start a discussion about PIP-31: Introduce Function
> >
> > As Paimon has supported table and view， maybe we could support function，
> > and we could use it in：
> > 1. Unified column-level filtering and processing capabilities (ps: for
> > email information encryption and decryption)
> > 2. Parameterized view capabilities: similar to retrieving views, it
> > supports parameters
> >
> > Looking forward to your feedback, thanks!
> > [1] pip-31
> > <
> https://cwiki.apache.org/confluence/display/PAIMON/PIP-31%3A+Introduce+Function
> >
> >
> > Best, Jerry
>
>
Re: Support Function
Posted by yejunhao <44...@qq.com.INVALID>.
Looks nice,
Is that means, we can define our UDF in paimon? And could you provide some Interface of the java?
And why trino only support sql function not jar function?
Best,
Junhao
> 2025年4月27日 11:38，lining jing <ji...@gmail.com> 写道：
>
> Hi devs,
> I would like to start a discussion about PIP-31: Introduce Function
>
> As Paimon has supported table and view， maybe we could support function，
> and we could use it in：
> 1. Unified column-level filtering and processing capabilities (ps: for
> email information encryption and decryption)
> 2. Parameterized view capabilities: similar to retrieving views, it
> supports parameters
>
> Looking forward to your feedback, thanks!
> [1] pip-31
> <https://cwiki.apache.org/confluence/display/PAIMON/PIP-31%3A+Introduce+Function>
>
> Best, Jerry
Re: Support Function
Posted by Jingsong Li <ji...@gmail.com>.
Thanks a lot.
Looks good to me now.
Best,
Jingsong
On Mon, Apr 28, 2025 at 11:47 AM lining jing <ji...@gmail.com> wrote:
>
> Thanks for your reply. I hava updates the pip by your suggestion.
>>
>>
>> From: Jingsong Li <ji...@gmail.com>
>> Date: Sun, Apr 27, 2025 at 12:41 PM
>> Subject: Re: Support Function
>> To: <de...@paimon.apache.org>
>>
>>
>> Thanks Jerry for starting this discussion.
>>
>> Functions are indeed important to us. We hope that engines will
>> support SQL functions in the future, but now, we cannot support a
>> unified function definition.
>>
>> ## Comments
>>
>> > parameters
>>
>> Name to be `inputParams`?
>>
>> Should it just be Array<DataField>?
>>
>> > returnType
>>
>> Name to be `returnParams`?
>>
>> It can also just be Array<DataField>.
>>
>> > properties
>>
>> Name to be `options`.
>>
>> > type and definitions
>>
>> We can just have a `Map<String, FunctionDefinition> definitions `
>> field, type to definition.
>>
>> Types can be:
>> 1. "sql" => SqlFunctionDefinition(String sql)
>> 2. "flink" => FlinkFunctionDefinition(String className,
>> FunctionLanguage language, List<ResourceUri> resources)
>> 3. "lambda" => LambdaFunctionDefinition(String definition,
>> FunctionLanguage language)
>>
>> ResourceUri should be similar to Flink.
>>
>> > catalog
>>
>> We don't need a catalog field in this class.
>>
>> ## Example
>>
>> You should have an example, for example, the same function is used
>> across engines by Flink and Spark.
>>
>> Best,
>> Jingsong
>>
>> On Sun, Apr 27, 2025 at 11:48 AM lining jing <ji...@gmail.com> wrote:
>> >
>> > Hi devs,
>> > I would like to start a discussion about PIP-31: Introduce Function
>> >
>> > As Paimon has supported table and view， maybe we could support function，
>> > and we could use it in：
>> > 1. Unified column-level filtering and processing capabilities (ps: for
>> > email information encryption and decryption)
>> > 2. Parameterized view capabilities: similar to retrieving views, it
>> > supports parameters
>> >
>> > Looking forward to your feedback, thanks!
>> > [1] pip-31
>> > <https://cwiki.apache.org/confluence/display/PAIMON/PIP-31%3A+Introduce+Function>
>> >
>> > Best, Jerry
Re: Support Function
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Jerry for starting this discussion.
Functions are indeed important to us. We hope that engines will
support SQL functions in the future, but now, we cannot support a
unified function definition.
## Comments
> parameters
Name to be `inputParams`?
Should it just be Array<DataField>?
> returnType
Name to be `returnParams`?
It can also just be Array<DataField>.
> properties
Name to be `options`.
> type and definitions
We can just have a `Map<String, FunctionDefinition> definitions `
field, type to definition.
Types can be:
1. "sql" => SqlFunctionDefinition(String sql)
2. "flink" => FlinkFunctionDefinition(String className,
FunctionLanguage language, List<ResourceUri> resources)
3. "lambda" => LambdaFunctionDefinition(String definition,
FunctionLanguage language)
ResourceUri should be similar to Flink.
> catalog
We don't need a catalog field in this class.
## Example
You should have an example, for example, the same function is used
across engines by Flink and Spark.
Best,
Jingsong
On Sun, Apr 27, 2025 at 11:48 AM lining jing <ji...@gmail.com> wrote:
>
> Hi devs,
> I would like to start a discussion about PIP-31: Introduce Function
>
> As Paimon has supported table and view， maybe we could support function，
> and we could use it in：
> 1. Unified column-level filtering and processing capabilities (ps: for
> email information encryption and decryption)
> 2. Parameterized view capabilities: similar to retrieving views, it
> supports parameters
>
> Looking forward to your feedback, thanks!
> [1] pip-31
> <https://cwiki.apache.org/confluence/display/PAIMON/PIP-31%3A+Introduce+Function>
>
> Best, Jerry