You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Mika Naylor <ma...@autophagy.io> on 2026/08/26 10:42:30 UTC
[DISCUSS] FLIP-609: Type Inference for Python User Defined Functions
Hey everyone!
I would like to kick off a discussion on FLIP-609: Type Inference for Python User Defined Functions[1].
While working with Python UDFs, I often noticed I was doing some duplicate effort around type hinting - in one place
so the planner knows the Flink specific input/output types of the UDF, and also on the function itself so I could have
an extra layer of checking my type assumptions/flow using type checking tools like mypy. I also noticed that there was
a bit of friction in doing this, especially since the form of specifying input types through the UDF constructor was
necessarily disconnected from the actual function arguments the types referred to.
This FLIP proposes to add a type hints -> Flink types inference layer for UDFs, so that users in ideal cases should
only have to annotate their function using native Python type hints, and we can infer the input/output Flink types from
those. In more complex cases, where users want to specify a specific Flink type rather than a Python type, I also propose
to add some shadow types that wrap the Flink types in a corresponding Python type, so that both type checking works,
and the Flink specific type hints are bound to the actual arguments, rather than just the argument positions via the udf
decorator. So that a user could do the following:
from dataclasses import dataclass
from typing import Optional
from pyflink.table import udf
from pyflink.table.typehints import TinyInt, SmallInt, Decimal
Money = Decimal(18, 2)
@dataclass
class PricingResult:
final_price: Money
discount_applied: bool
tier: TinyInt
@udf()
def apply_discount(
price: Money,
discount_pct: Optional[SmallInt],
tier: TinyInt,
) -> PricingResult:
pct = discount_pct or 0
discount = price * pct / 100
return PricingResult(
final_price=price - discount,
discount_applied=pct > 0,
tier=tier,
)
Would love any thoughts or feedback the community might have on this proposal!
Kind regards,
Mika Naylor
[1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286339/FLIP-609+Type+Inference+for+Python+User+Defined+Functions
Re: [DISCUSS] FLIP-609: Type Inference for Python User Defined Functions
Posted by Dian Fu <di...@gmail.com>.
Hi Mika,
Thanks for this great work! The FLIP LGTM overall. It would be a great
improvement.
Regards,
Dian
On Thu, Sep 3, 2026 at 6:38 PM Mika Naylor <ma...@autophagy.io> wrote:
>
> Hi all!
>
> Thank you for the discussion on this. Just wanted to bump the discussion again in case anyone had any
> input or feedback (especially around the deprecation plan in the FLIP) - I would like to try and open
> voting on it later this week :)
>
> Kind regards,
> Mika
>
> > On 26. Aug 2026, at 12:42, Mika Naylor <ma...@autophagy.io> wrote:
> >
> > Hey everyone!
> >
> > I would like to kick off a discussion on FLIP-609: Type Inference for Python User Defined Functions[1].
> >
> > While working with Python UDFs, I often noticed I was doing some duplicate effort around type hinting - in one place
> > so the planner knows the Flink specific input/output types of the UDF, and also on the function itself so I could have
> > an extra layer of checking my type assumptions/flow using type checking tools like mypy. I also noticed that there was
> > a bit of friction in doing this, especially since the form of specifying input types through the UDF constructor was
> > necessarily disconnected from the actual function arguments the types referred to.
> >
> > This FLIP proposes to add a type hints -> Flink types inference layer for UDFs, so that users in ideal cases should
> > only have to annotate their function using native Python type hints, and we can infer the input/output Flink types from
> > those. In more complex cases, where users want to specify a specific Flink type rather than a Python type, I also propose
> > to add some shadow types that wrap the Flink types in a corresponding Python type, so that both type checking works,
> > and the Flink specific type hints are bound to the actual arguments, rather than just the argument positions via the udf
> > decorator. So that a user could do the following:
> >
> > from dataclasses import dataclass
> > from typing import Optional
> > from pyflink.table import udf
> > from pyflink.table.typehints import TinyInt, SmallInt, Decimal
> >
> > Money = Decimal(18, 2)
> >
> > @dataclass
> > class PricingResult:
> > final_price: Money
> > discount_applied: bool
> > tier: TinyInt
> >
> > @udf()
> > def apply_discount(
> > price: Money,
> > discount_pct: Optional[SmallInt],
> > tier: TinyInt,
> > ) -> PricingResult:
> > pct = discount_pct or 0
> > discount = price * pct / 100
> > return PricingResult(
> > final_price=price - discount,
> > discount_applied=pct > 0,
> > tier=tier,
> > )
> >
> > Would love any thoughts or feedback the community might have on this proposal!
> >
> > Kind regards,
> > Mika Naylor
> >
> > [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286339/FLIP-609+Type+Inference+for+Python+User+Defined+Functions
> >
> >
>
Re: [DISCUSS] FLIP-609: Type Inference for Python User Defined Functions
Posted by Mika Naylor <ma...@autophagy.io>.
Hi all!
Thank you for the discussion on this. Just wanted to bump the discussion again in case anyone had any
input or feedback (especially around the deprecation plan in the FLIP) - I would like to try and open
voting on it later this week :)
Kind regards,
Mika
> On 26. Aug 2026, at 12:42, Mika Naylor <ma...@autophagy.io> wrote:
>
> Hey everyone!
>
> I would like to kick off a discussion on FLIP-609: Type Inference for Python User Defined Functions[1].
>
> While working with Python UDFs, I often noticed I was doing some duplicate effort around type hinting - in one place
> so the planner knows the Flink specific input/output types of the UDF, and also on the function itself so I could have
> an extra layer of checking my type assumptions/flow using type checking tools like mypy. I also noticed that there was
> a bit of friction in doing this, especially since the form of specifying input types through the UDF constructor was
> necessarily disconnected from the actual function arguments the types referred to.
>
> This FLIP proposes to add a type hints -> Flink types inference layer for UDFs, so that users in ideal cases should
> only have to annotate their function using native Python type hints, and we can infer the input/output Flink types from
> those. In more complex cases, where users want to specify a specific Flink type rather than a Python type, I also propose
> to add some shadow types that wrap the Flink types in a corresponding Python type, so that both type checking works,
> and the Flink specific type hints are bound to the actual arguments, rather than just the argument positions via the udf
> decorator. So that a user could do the following:
>
> from dataclasses import dataclass
> from typing import Optional
> from pyflink.table import udf
> from pyflink.table.typehints import TinyInt, SmallInt, Decimal
>
> Money = Decimal(18, 2)
>
> @dataclass
> class PricingResult:
> final_price: Money
> discount_applied: bool
> tier: TinyInt
>
> @udf()
> def apply_discount(
> price: Money,
> discount_pct: Optional[SmallInt],
> tier: TinyInt,
> ) -> PricingResult:
> pct = discount_pct or 0
> discount = price * pct / 100
> return PricingResult(
> final_price=price - discount,
> discount_applied=pct > 0,
> tier=tier,
> )
>
> Would love any thoughts or feedback the community might have on this proposal!
>
> Kind regards,
> Mika Naylor
>
> [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286339/FLIP-609+Type+Inference+for+Python+User+Defined+Functions
>
>
Re: [DISCUSS] FLIP-609: Type Inference for Python User Defined Functions
Posted by Mika Naylor <ma...@autophagy.io>.
Hi Timo,
Yeah, that’s a cool idea. Hopefully if this kind of type inference proves useful enough and captures
enough of the type usage people need, it could be extended to elsewhere to make working with
types in PyFlink feel a bit more natural (unless you need to be specific and granular).
Mika
> On 31. Aug 2026, at 17:47, Timo Theusner via dev <de...@flink.apache.org> wrote:
>
> Thank you Mika for the FLIP! I like the proposal and it would improve the
> current PyFlink experience.
> As a follow-up, we could think about other use cases for the type hints.
> For example, when building Schemas importing `pyflink.table.typehints` and
> `pyflink.table.DataTypes` might be confusing.
>
> Br,
> Timo
>
> On Fri, Aug 28, 2026 at 4:17 AM Zander Matheson <a.w.matheson@gmail.com <ma...@gmail.com>>
> wrote:
>
>> This would be a significantly improved devex and I love the idea! Thanks
>> for proposing, Mika.
>>
>> On Wed, Aug 26, 2026 at 3:45 AM Mika Naylor <ma...@autophagy.io> wrote:
>>
>>> Hey everyone!
>>>
>>> I would like to kick off a discussion on FLIP-609: Type Inference for
>>> Python User Defined Functions[1].
>>>
>>> While working with Python UDFs, I often noticed I was doing some
>> duplicate
>>> effort around type hinting - in one place
>>> so the planner knows the Flink specific input/output types of the UDF,
>> and
>>> also on the function itself so I could have
>>> an extra layer of checking my type assumptions/flow using type checking
>>> tools like mypy. I also noticed that there was
>>> a bit of friction in doing this, especially since the form of specifying
>>> input types through the UDF constructor was
>>> necessarily disconnected from the actual function arguments the types
>>> referred to.
>>>
>>> This FLIP proposes to add a type hints -> Flink types inference layer for
>>> UDFs, so that users in ideal cases should
>>> only have to annotate their function using native Python type hints, and
>>> we can infer the input/output Flink types from
>>> those. In more complex cases, where users want to specify a specific
>> Flink
>>> type rather than a Python type, I also propose
>>> to add some shadow types that wrap the Flink types in a corresponding
>>> Python type, so that both type checking works,
>>> and the Flink specific type hints are bound to the actual arguments,
>>> rather than just the argument positions via the udf
>>> decorator. So that a user could do the following:
>>>
>>> from dataclasses import dataclass
>>> from typing import Optional
>>> from pyflink.table import udf
>>> from pyflink.table.typehints import TinyInt, SmallInt, Decimal
>>>
>>> Money = Decimal(18, 2)
>>>
>>> @dataclass
>>> class PricingResult:
>>> final_price: Money
>>> discount_applied: bool
>>> tier: TinyInt
>>>
>>> @udf()
>>> def apply_discount(
>>> price: Money,
>>> discount_pct: Optional[SmallInt],
>>> tier: TinyInt,
>>> ) -> PricingResult:
>>> pct = discount_pct or 0
>>> discount = price * pct / 100
>>> return PricingResult(
>>> final_price=price - discount,
>>> discount_applied=pct > 0,
>>> tier=tier,
>>> )
>>>
>>> Would love any thoughts or feedback the community might have on this
>>> proposal!
>>>
>>> Kind regards,
>>> Mika Naylor
>>>
>>> [1]
>>>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286339/FLIP-609*Type*Inference*for*Python*User*Defined*Functions__;KysrKysrKw!!Ayb5sqE7!pS7rgUV6JteB7ZRfw1GM-bBQkZZJPcmPH8jGatn0u52fueih7VIxW_v3HuZxne1FGTT7Uh21ofDzmpPtOU1IX01Dig$
>>>
>>>
>>>
>>
>
>
> --
>
> Timo Theusner
>
> Senior Software Engineer
>
> ttheusner@confluent.io <ma...@confluent.io>
>
> <https://confluent.io <https://confluent.io/>>
>
> Follow us: [image: RSS]
> <https://www.confluent.io/blog/?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image:
> LinkedIn] <https://www.linkedin.com/company/confluent/posts/?feedView=all>[image:
> YouTube] <ht...@Confluent>[image: Confluent Slack
> Community]
> <https://confluentcommunity.slack.com/join/shared_invite/zt-3az13rpkx-CMUmebJjiFwCn65VOaSEPA#/shared-invite/email>
>
> <https://confluent.io/data-monster>
Re: [DISCUSS] FLIP-609: Type Inference for Python User Defined Functions
Posted by Timo Theusner via dev <de...@flink.apache.org>.
Thank you Mika for the FLIP! I like the proposal and it would improve the
current PyFlink experience.
As a follow-up, we could think about other use cases for the type hints.
For example, when building Schemas importing `pyflink.table.typehints` and
`pyflink.table.DataTypes` might be confusing.
Br,
Timo
On Fri, Aug 28, 2026 at 4:17 AM Zander Matheson <a....@gmail.com>
wrote:
> This would be a significantly improved devex and I love the idea! Thanks
> for proposing, Mika.
>
> On Wed, Aug 26, 2026 at 3:45 AM Mika Naylor <ma...@autophagy.io> wrote:
>
> > Hey everyone!
> >
> > I would like to kick off a discussion on FLIP-609: Type Inference for
> > Python User Defined Functions[1].
> >
> > While working with Python UDFs, I often noticed I was doing some
> duplicate
> > effort around type hinting - in one place
> > so the planner knows the Flink specific input/output types of the UDF,
> and
> > also on the function itself so I could have
> > an extra layer of checking my type assumptions/flow using type checking
> > tools like mypy. I also noticed that there was
> > a bit of friction in doing this, especially since the form of specifying
> > input types through the UDF constructor was
> > necessarily disconnected from the actual function arguments the types
> > referred to.
> >
> > This FLIP proposes to add a type hints -> Flink types inference layer for
> > UDFs, so that users in ideal cases should
> > only have to annotate their function using native Python type hints, and
> > we can infer the input/output Flink types from
> > those. In more complex cases, where users want to specify a specific
> Flink
> > type rather than a Python type, I also propose
> > to add some shadow types that wrap the Flink types in a corresponding
> > Python type, so that both type checking works,
> > and the Flink specific type hints are bound to the actual arguments,
> > rather than just the argument positions via the udf
> > decorator. So that a user could do the following:
> >
> > from dataclasses import dataclass
> > from typing import Optional
> > from pyflink.table import udf
> > from pyflink.table.typehints import TinyInt, SmallInt, Decimal
> >
> > Money = Decimal(18, 2)
> >
> > @dataclass
> > class PricingResult:
> > final_price: Money
> > discount_applied: bool
> > tier: TinyInt
> >
> > @udf()
> > def apply_discount(
> > price: Money,
> > discount_pct: Optional[SmallInt],
> > tier: TinyInt,
> > ) -> PricingResult:
> > pct = discount_pct or 0
> > discount = price * pct / 100
> > return PricingResult(
> > final_price=price - discount,
> > discount_applied=pct > 0,
> > tier=tier,
> > )
> >
> > Would love any thoughts or feedback the community might have on this
> > proposal!
> >
> > Kind regards,
> > Mika Naylor
> >
> > [1]
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286339/FLIP-609*Type*Inference*for*Python*User*Defined*Functions__;KysrKysrKw!!Ayb5sqE7!pS7rgUV6JteB7ZRfw1GM-bBQkZZJPcmPH8jGatn0u52fueih7VIxW_v3HuZxne1FGTT7Uh21ofDzmpPtOU1IX01Dig$
> >
> >
> >
>
--
Timo Theusner
Senior Software Engineer
ttheusner@confluent.io
<https://confluent.io>
Follow us: [image: RSS]
<https://www.confluent.io/blog/?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image:
LinkedIn] <https://www.linkedin.com/company/confluent/posts/?feedView=all>[image:
YouTube] <ht...@Confluent>[image: Confluent Slack
Community]
<https://confluentcommunity.slack.com/join/shared_invite/zt-3az13rpkx-CMUmebJjiFwCn65VOaSEPA#/shared-invite/email>
<https://confluent.io/data-monster>
Re: [DISCUSS] FLIP-609: Type Inference for Python User Defined Functions
Posted by Zander Matheson <a....@gmail.com>.
This would be a significantly improved devex and I love the idea! Thanks
for proposing, Mika.
On Wed, Aug 26, 2026 at 3:45 AM Mika Naylor <ma...@autophagy.io> wrote:
> Hey everyone!
>
> I would like to kick off a discussion on FLIP-609: Type Inference for
> Python User Defined Functions[1].
>
> While working with Python UDFs, I often noticed I was doing some duplicate
> effort around type hinting - in one place
> so the planner knows the Flink specific input/output types of the UDF, and
> also on the function itself so I could have
> an extra layer of checking my type assumptions/flow using type checking
> tools like mypy. I also noticed that there was
> a bit of friction in doing this, especially since the form of specifying
> input types through the UDF constructor was
> necessarily disconnected from the actual function arguments the types
> referred to.
>
> This FLIP proposes to add a type hints -> Flink types inference layer for
> UDFs, so that users in ideal cases should
> only have to annotate their function using native Python type hints, and
> we can infer the input/output Flink types from
> those. In more complex cases, where users want to specify a specific Flink
> type rather than a Python type, I also propose
> to add some shadow types that wrap the Flink types in a corresponding
> Python type, so that both type checking works,
> and the Flink specific type hints are bound to the actual arguments,
> rather than just the argument positions via the udf
> decorator. So that a user could do the following:
>
> from dataclasses import dataclass
> from typing import Optional
> from pyflink.table import udf
> from pyflink.table.typehints import TinyInt, SmallInt, Decimal
>
> Money = Decimal(18, 2)
>
> @dataclass
> class PricingResult:
> final_price: Money
> discount_applied: bool
> tier: TinyInt
>
> @udf()
> def apply_discount(
> price: Money,
> discount_pct: Optional[SmallInt],
> tier: TinyInt,
> ) -> PricingResult:
> pct = discount_pct or 0
> discount = price * pct / 100
> return PricingResult(
> final_price=price - discount,
> discount_applied=pct > 0,
> tier=tier,
> )
>
> Would love any thoughts or feedback the community might have on this
> proposal!
>
> Kind regards,
> Mika Naylor
>
> [1]
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286339/FLIP-609+Type+Inference+for+Python+User+Defined+Functions
>
>
>