You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Abdul Qadeer <qu...@gmail.com> on 2018/10/30 06:00:24 UTC

Watermark Propagation using SDF

Hi!

I would like to know if watermarks can be updated from SDFs and propagated
to other operators for windowing? As per the design docs [1] [2] it seems
plausible to me as the idea behind SDFs is to replace the Source APIs and
make a DoFn more generic.

[1] Splittable DoFn Proposal
<https://docs.google.com/document/d/1AQmx-T9XjSi1PNoEp5_L-lT0j7BkgTbmQnc6uFEMI4c/edit#>
[2] SDF API Changes
<https://docs.google.com/document/d/1BGc8pM1GOvZhwR9SARSVte-20XEoBUxrGJ5gTWXdv3c/edit>

Re: Watermark Propagation using SDF

Posted by Lukasz Cwik <lc...@google.com>.

Yes, reporting watermarks is a key concept within SDFs and needed for
downstream windowing operations.

On Mon, Oct 29, 2018 at 11:00 PM Abdul Qadeer <qu...@gmail.com> wrote:

> Hi!
>
> I would like to know if watermarks can be updated from SDFs and propagated
> to other operators for windowing? As per the design docs [1] [2] it seems
> plausible to me as the idea behind SDFs is to replace the Source APIs and
> make a DoFn more generic.
>
> [1] Splittable DoFn Proposal
> <https://docs.google.com/document/d/1AQmx-T9XjSi1PNoEp5_L-lT0j7BkgTbmQnc6uFEMI4c/edit#>
> [2] SDF API Changes
> <https://docs.google.com/document/d/1BGc8pM1GOvZhwR9SARSVte-20XEoBUxrGJ5gTWXdv3c/edit>
>
>