You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pulsar.apache.org by Eron Wright <ew...@streamnative.io.INVALID> on 2021/10/02 20:40:49 UTC

[PIP 102] Pulsar Watermarking

Hi,
I've created PIP 102 to introduce the concept of watermarking into Pulsar,
to improve the correctness of stream processing applications that use
Pulsar topics as a source and/or sink.

https://github.com/apache/pulsar/issues/12267

Some initial efforts were made during the 2021 Apache Pulsar Hackathon to
demonstrate the value proposition.  Here's some relevant links:

https://github.com/EronWright/pulsar-hackathon-2021-projects-watermarking

https://streamnative.io/blog/community/2021-06-22-pulsar-hackathon-2021-winners-announced/

I'm giving a talk at Pulsar Summit Europe on this idea, and will post more
materials as they become available:
https://pulsar-summit.org/en/event/europe-2021/sessions/pulsar-watermarking

Looking forward to discussing this idea with the Pulsar community!
Eron Wright
StreamNative

Re: [PIP 102] Pulsar Watermarking

Posted by Eron Wright <ew...@streamnative.io.INVALID>.

Thanks for the feedback, I will address it along with feedback from the
upcoming summit, and then post an updated draft.   I also have details
about evolving the Flink connector and Pulsar function framework to support
this feature.

On Sun, Oct 3, 2021 at 4:57 AM Enrico Olivelli <eo...@gmail.com> wrote:

> Great work!
>
> Overall the proposal looks good to me.
> But I think we are missing these topics:
> - how to enable this feature (new broker configuration flag?)
> - how to handle old consumer connecting to a system with watermarks (we
> should not deliver the watermark events?)
> - tooling to support operations (reset/inspect...)
>
>
> Thanks for contributing this precious feature that is missing in Pulsar.
>
> Best regards
> Enrico
>
> Il Sab 2 Ott 2021, 22:41 Eron Wright <ew...@streamnative.io.invalid> ha
> scritto:
>
> > Hi,
> > I've created PIP 102 to introduce the concept of watermarking into
> Pulsar,
> > to improve the correctness of stream processing applications that use
> > Pulsar topics as a source and/or sink.
> >
> > https://github.com/apache/pulsar/issues/12267
> >
> > Some initial efforts were made during the 2021 Apache Pulsar Hackathon to
> > demonstrate the value proposition.  Here's some relevant links:
> >
> >
> https://github.com/EronWright/pulsar-hackathon-2021-projects-watermarking
> >
> >
> >
> https://streamnative.io/blog/community/2021-06-22-pulsar-hackathon-2021-winners-announced/
> >
> > I'm giving a talk at Pulsar Summit Europe on this idea, and will post
> more
> > materials as they become available:
> >
> https://pulsar-summit.org/en/event/europe-2021/sessions/pulsar-watermarking
> >
> > Looking forward to discussing this idea with the Pulsar community!
> > Eron Wright
> > StreamNative
> >
>

Re: [PIP 102] Pulsar Watermarking

Posted by Enrico Olivelli <eo...@gmail.com>.

Great work!

Overall the proposal looks good to me.
But I think we are missing these topics:
- how to enable this feature (new broker configuration flag?)
- how to handle old consumer connecting to a system with watermarks (we
should not deliver the watermark events?)
- tooling to support operations (reset/inspect...)


Thanks for contributing this precious feature that is missing in Pulsar.

Best regards
Enrico

Il Sab 2 Ott 2021, 22:41 Eron Wright <ew...@streamnative.io.invalid> ha
scritto:

> Hi,
> I've created PIP 102 to introduce the concept of watermarking into Pulsar,
> to improve the correctness of stream processing applications that use
> Pulsar topics as a source and/or sink.
>
> https://github.com/apache/pulsar/issues/12267
>
> Some initial efforts were made during the 2021 Apache Pulsar Hackathon to
> demonstrate the value proposition.  Here's some relevant links:
>
> https://github.com/EronWright/pulsar-hackathon-2021-projects-watermarking
>
>
> https://streamnative.io/blog/community/2021-06-22-pulsar-hackathon-2021-winners-announced/
>
> I'm giving a talk at Pulsar Summit Europe on this idea, and will post more
> materials as they become available:
> https://pulsar-summit.org/en/event/europe-2021/sessions/pulsar-watermarking
>
> Looking forward to discussing this idea with the Pulsar community!
> Eron Wright
> StreamNative
>