You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Rui Fu <rf...@apache.org> on 2023/03/01 03:16:54 UTC

Re: [DISCUSS] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

+1

Best,

Rui Fu
On Feb 28, 2023 at 05:35 +0800, laminar <tp...@gmail.com>, wrote:
> Hi all,
>
> I would like to discuss supporting the processingGuarantess `EFFECTIVELY_ONCE` in python function runtime.
>
> In this PR's(https://github.com/apache/pulsar/pull/18929) discussion, we conclude that to achieve the exactly processing guarantees for `EFFECTIVELY_ONCE`, the user needs to ensure that the following pre-requisites are met.
>
> 1. deduplication is enabled
> 2. set ProcessingGuarantees to EFFECTIVELY_ONCE
> 3. the function has only one source topic and one sink topic (both are non-partitioned)
> 4. if partitioned topic is enabled, ensure that the number of partitions (of both source and sink topics) is the same
>
> Currently, neither the python function runtime nor the java function runtime can support the `EFFECTIVELY_ONCE` processing guarantee when using partitioned topics.
>
> So in order to make python functions support `EFFECTIVELY_ONCE` processing guarantee, I think we can introduce this feature incrementally, i.e. support the `EFFECTIVELY_ONCE` processing guarantee for non-partitioned topics first.
>
> Then follow up with Rui’s suggestion(https://github.com/apache/pulsar/pull/18929#issuecomment-1445977320) to improve this feature.
>
>

Re: [DISCUSS] Support processingGuarantees "EFFECTIVELY_ONCE" in python function

Posted by Enrico Olivelli <eo...@gmail.com>.
Perfect

Thanks
Enrico

Il giorno mer 1 mar 2023 alle ore 04:17 Rui Fu <rf...@apache.org> ha scritto:
>
> +1
>
> Best,
>
> Rui Fu
> On Feb 28, 2023 at 05:35 +0800, laminar <tp...@gmail.com>, wrote:
> > Hi all,
> >
> > I would like to discuss supporting the processingGuarantess `EFFECTIVELY_ONCE` in python function runtime.
> >
> > In this PR's(https://github.com/apache/pulsar/pull/18929) discussion, we conclude that to achieve the exactly processing guarantees for `EFFECTIVELY_ONCE`, the user needs to ensure that the following pre-requisites are met.
> >
> > 1. deduplication is enabled
> > 2. set ProcessingGuarantees to EFFECTIVELY_ONCE
> > 3. the function has only one source topic and one sink topic (both are non-partitioned)
> > 4. if partitioned topic is enabled, ensure that the number of partitions (of both source and sink topics) is the same
> >
> > Currently, neither the python function runtime nor the java function runtime can support the `EFFECTIVELY_ONCE` processing guarantee when using partitioned topics.
> >
> > So in order to make python functions support `EFFECTIVELY_ONCE` processing guarantee, I think we can introduce this feature incrementally, i.e. support the `EFFECTIVELY_ONCE` processing guarantee for non-partitioned topics first.
> >
> > Then follow up with Rui’s suggestion(https://github.com/apache/pulsar/pull/18929#issuecomment-1445977320) to improve this feature.
> >
> >