Side windows and late inputs

[Raman Gupta <ro...@gmail.com>]

I have a topology that looks like this:

input -> 1hr window -> view
      |
      -> 5m window -> dofn(view)

IOW -- an input is windowed into both 1 hour windows, as well as 5 minute
windows. The 1 hour window data is collected into a view.

Meanwhile, the input is also windowed with 5 minute windows, there is some
processing, and then finally a DoFn that uses the view as a side input.

Both windows allow late data, and fire immediately on a late element. My
expectation would have been that any processing occurring in the DoFn after
the processing time of the late element would see the late element in the
view. However, that doesn't seem to be the case. Why?

Is there another approach I can use that doesn't suffer from this problem?

Regards,
Raman Gupta

Re: Side windows and late inputs

[Raman Gupta <ro...@gmail.com>]

I'm also having the opposite problem -- my unit test moves the watermark to
just before a timestamped input. I was expecting the DoFn to execute for
all windows up to that watermark, seeing nothing in the side input.

However, that isn't the case -- the test shows that the DoFn side input
*before* the watermark, contains the input timestamped *after* the
watermark. This makes no sense to me.

Regards,
Raman


On Sat, Apr 10, 2021 at 2:46 AM Raman Gupta <ro...@gmail.com> wrote:

> I have a topology that looks like this:
>
> input -> 1hr window -> view
>       |
>       -> 5m window -> dofn(view)
>
> IOW -- an input is windowed into both 1 hour windows, as well as 5 minute
> windows. The 1 hour window data is collected into a view.
>
> Meanwhile, the input is also windowed with 5 minute windows, there is some
> processing, and then finally a DoFn that uses the view as a side input.
>
> Both windows allow late data, and fire immediately on a late element. My
> expectation would have been that any processing occurring in the DoFn after
> the processing time of the late element would see the late element in the
> view. However, that doesn't seem to be the case. Why?
>
> Is there another approach I can use that doesn't suffer from this problem?
>
> Regards,
> Raman Gupta
>
>