Off-by-one issues in the windowing code

[Márton Balassi <ba...@gmail.com>]

I have found two off-by-one issues in the windowing code.

The first may result in duplicate data in the last window and is easy to
fix. [1]

The second may result data being swallowed in the last window, and is also
not difficult to fix. [2]

I've talked to Aljoscha about fixing the second one, and he suggested not
to fix it right away as that part should be rewritten soon anyways, maybe
we remove that functionality as a whole. As this is also in the 0.9.0
release I would still opt for having it, at least for the sake of 0.9.1.

What do you think?

[1] https://issues.apache.org/jira/browse/FLINK-2285
[2] https://issues.apache.org/jira/browse/FLINK-2286

Best,

Marton

Re: Off-by-one issues in the windowing code

[Maximilian Michels <mx...@apache.org>]

Hey Marton,

Thanks for spotting the two issues and fixing the first one. Those should
definitely be fixed in 0.9.1.

Cheers,
Max

On Mon, Jun 29, 2015 at 11:14 AM, Gyula Fóra <gy...@gmail.com> wrote:

> The second issue is related to parallel time based aggregations. I think we
> should fix this for 0.9.1.
>
> Also since the fix as you said is rather straight-forward there is no harm
> doing it. As I understand if we keep the functionality of having time based
> global windows, the implementations for merging the partial aggregates will
> be conceptually similar after Aljoscha's update as well.
>
> Márton Balassi <ba...@gmail.com> ezt írta (időpont: 2015. jún.
> 29., H, 11:02):
>
> > I have found two off-by-one issues in the windowing code.
> >
> > The first may result in duplicate data in the last window and is easy to
> > fix. [1]
> >
> > The second may result data being swallowed in the last window, and is
> also
> > not difficult to fix. [2]
> >
> > I've talked to Aljoscha about fixing the second one, and he suggested not
> > to fix it right away as that part should be rewritten soon anyways, maybe
> > we remove that functionality as a whole. As this is also in the 0.9.0
> > release I would still opt for having it, at least for the sake of 0.9.1.
> >
> > What do you think?
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-2285
> > [2] https://issues.apache.org/jira/browse/FLINK-2286
> >
> > Best,
> >
> > Marton
> >
>

Re: Off-by-one issues in the windowing code

[Gyula Fóra <gy...@gmail.com>]

The second issue is related to parallel time based aggregations. I think we
should fix this for 0.9.1.

Also since the fix as you said is rather straight-forward there is no harm
doing it. As I understand if we keep the functionality of having time based
global windows, the implementations for merging the partial aggregates will
be conceptually similar after Aljoscha's update as well.

Márton Balassi <ba...@gmail.com> ezt írta (időpont: 2015. jún.
29., H, 11:02):

> I have found two off-by-one issues in the windowing code.
>
> The first may result in duplicate data in the last window and is easy to
> fix. [1]
>
> The second may result data being swallowed in the last window, and is also
> not difficult to fix. [2]
>
> I've talked to Aljoscha about fixing the second one, and he suggested not
> to fix it right away as that part should be rewritten soon anyways, maybe
> we remove that functionality as a whole. As this is also in the 0.9.0
> release I would still opt for having it, at least for the sake of 0.9.1.
>
> What do you think?
>
> [1] https://issues.apache.org/jira/browse/FLINK-2285
> [2] https://issues.apache.org/jira/browse/FLINK-2286
>
> Best,
>
> Marton
>