python AfterCount() trigger behavior not aligned with java

[Leiyi Zhang <le...@google.com>]

Hi everyone!
I noticed that the behavior of AfterCount() trigger seems to be different
between python sdk and the java one, so I created a few tests to show the
difference, but in general I think the python sdk will buffer on result
instead of input elements.

What do you guys think?

and here are the tests. I ran them in batch mode.

Sincerely,
Leiyi

Re: python AfterCount() trigger behavior not aligned with java

[Leiyi Zhang <le...@google.com>]

Thank you Robert!

On Mon, Aug 17, 2020 at 5:52 PM Robert Bradshaw <ro...@google.com> wrote:

> Correct, everything is per-key. To allow triggering after n events you
> would have to given them all the same key. (Note that this would
> potentially introduce a bottleneck, as they would all be shuffled to
> the same machine.)
>
> On Mon, Aug 17, 2020 at 4:01 PM Leiyi Zhang <le...@google.com> wrote:
> >
> > Thank you for explaining the details Robert!
> >
> > I do have 1 more question: is there a way in beam that allows triggering
> after n events arrived at the gbk step from previous parts of a pipeline?
> because afterCount() trigger for gbk is per-key, then it cannot be used
> here right?
> >
> > On Mon, Aug 17, 2020 at 3:22 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>
> >> Yeah, this is another subtlety. There's a notion of "window garbage
> >> collection" that's distinct from "window closing." Garbage collection
> >> happens regardless of whether the trigger was set iff the window is
> >> non-empty when the watermark + allowed lateness exceeds the end of
> >> window. (Well, there's a flag to control this behavior.) This has
> >> changed over time and I think you've uncovered a bug that the two SDKs
> >> are not consistent here.
> >>
> >> On Mon, Aug 17, 2020 at 2:29 PM Leiyi Zhang <le...@google.com> wrote:
> >> >
> >> > is it just for the python sdk or for both python and java sdk?
> >> > seems like java sdk will output result even if there are less than 3
> elements per key.
> >> >
> >> > On Mon, Aug 17, 2020 at 2:20 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >> >>
> >> >> Yes, GBK is non-determanistic in the face of triggers as well. All
> >> >> triggers are per-key, evaluated independently for each key, so it'd
> be
> >> >> "do I have at least 3 results for this key."
> >> >>
> >> >> On Mon, Aug 17, 2020 at 2:08 PM Leiyi Zhang <le...@google.com>
> wrote:
> >> >> >
> >> >> > Thank you very much for the reply,
> >> >> > Is the result of gbk non-deterministic as well? between "do I have
> at least 3 results PER KEY" vs "do I have at least 3 incoming events before
> I trigger GBK"
> >> >> >
> >> >> > On Mon, Aug 17, 2020 at 1:39 PM Robert Bradshaw <
> robertwb@google.com> wrote:
> >> >> >>
> >> >> >> Triggers in beam are non-determanistic; both behaviors are
> acceptable
> >> >> >> (especially for batch mode). In practice, production runners
> evaluate
> >> >> >> triggers (e.g. in this case "do I have at least two elements")
> whenver
> >> >> >> a new batch of data comes in (for the Python direct runner, in
> batch
> >> >> >> mode, all the data comes in at once). To have more control over
> this
> >> >> >> you can use TestPipeline, which will attempt to fire triggers as
> >> >> >> eagerly as possible.
> >> >> >>
> >> >> >> On Mon, Aug 17, 2020 at 1:16 PM Leiyi Zhang <le...@google.com>
> wrote:
> >> >> >> >
> >> >> >> > for GBK wtih AfterCount(3), java sdk results in this and python
> sdk results in this
> >> >> >> >
> >> >> >> > for global count with aftercount(2), java sdk results in 2 2 2
> 2 and python sdk results in 8.
> >> >> >> >
> >> >> >> > On Mon, Aug 17, 2020 at 12:40 PM Robert Bradshaw <
> robertwb@google.com> wrote:
> >> >> >> >>
> >> >> >> >> What results do you get in each?
> >> >> >> >>
> >> >> >> >> On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <
> leiyiz@google.com> wrote:
> >> >> >> >> >
> >> >> >> >> > Hi everyone!
> >> >> >> >> > I noticed that the behavior of AfterCount() trigger seems to
> be different between python sdk and the java one, so I created a few tests
> to show the difference, but in general I think the python sdk will buffer
> on result instead of input elements.
> >> >> >> >> >
> >> >> >> >> > What do you guys think?
> >> >> >> >> >
> >> >> >> >> > and here are the tests. I ran them in batch mode.
> >> >> >> >> >
> >> >> >> >> > Sincerely,
> >> >> >> >> > Leiyi
>

Re: python AfterCount() trigger behavior not aligned with java

[Robert Bradshaw <ro...@google.com>]

Correct, everything is per-key. To allow triggering after n events you
would have to given them all the same key. (Note that this would
potentially introduce a bottleneck, as they would all be shuffled to
the same machine.)

On Mon, Aug 17, 2020 at 4:01 PM Leiyi Zhang <le...@google.com> wrote:
>
> Thank you for explaining the details Robert!
>
> I do have 1 more question: is there a way in beam that allows triggering after n events arrived at the gbk step from previous parts of a pipeline? because afterCount() trigger for gbk is per-key, then it cannot be used here right?
>
> On Mon, Aug 17, 2020 at 3:22 PM Robert Bradshaw <ro...@google.com> wrote:
>>
>> Yeah, this is another subtlety. There's a notion of "window garbage
>> collection" that's distinct from "window closing." Garbage collection
>> happens regardless of whether the trigger was set iff the window is
>> non-empty when the watermark + allowed lateness exceeds the end of
>> window. (Well, there's a flag to control this behavior.) This has
>> changed over time and I think you've uncovered a bug that the two SDKs
>> are not consistent here.
>>
>> On Mon, Aug 17, 2020 at 2:29 PM Leiyi Zhang <le...@google.com> wrote:
>> >
>> > is it just for the python sdk or for both python and java sdk?
>> > seems like java sdk will output result even if there are less than 3 elements per key.
>> >
>> > On Mon, Aug 17, 2020 at 2:20 PM Robert Bradshaw <ro...@google.com> wrote:
>> >>
>> >> Yes, GBK is non-determanistic in the face of triggers as well. All
>> >> triggers are per-key, evaluated independently for each key, so it'd be
>> >> "do I have at least 3 results for this key."
>> >>
>> >> On Mon, Aug 17, 2020 at 2:08 PM Leiyi Zhang <le...@google.com> wrote:
>> >> >
>> >> > Thank you very much for the reply,
>> >> > Is the result of gbk non-deterministic as well? between "do I have at least 3 results PER KEY" vs "do I have at least 3 incoming events before I trigger GBK"
>> >> >
>> >> > On Mon, Aug 17, 2020 at 1:39 PM Robert Bradshaw <ro...@google.com> wrote:
>> >> >>
>> >> >> Triggers in beam are non-determanistic; both behaviors are acceptable
>> >> >> (especially for batch mode). In practice, production runners evaluate
>> >> >> triggers (e.g. in this case "do I have at least two elements") whenver
>> >> >> a new batch of data comes in (for the Python direct runner, in batch
>> >> >> mode, all the data comes in at once). To have more control over this
>> >> >> you can use TestPipeline, which will attempt to fire triggers as
>> >> >> eagerly as possible.
>> >> >>
>> >> >> On Mon, Aug 17, 2020 at 1:16 PM Leiyi Zhang <le...@google.com> wrote:
>> >> >> >
>> >> >> > for GBK wtih AfterCount(3), java sdk results in this and python sdk results in this
>> >> >> >
>> >> >> > for global count with aftercount(2), java sdk results in 2 2 2 2 and python sdk results in 8.
>> >> >> >
>> >> >> > On Mon, Aug 17, 2020 at 12:40 PM Robert Bradshaw <ro...@google.com> wrote:
>> >> >> >>
>> >> >> >> What results do you get in each?
>> >> >> >>
>> >> >> >> On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <le...@google.com> wrote:
>> >> >> >> >
>> >> >> >> > Hi everyone!
>> >> >> >> > I noticed that the behavior of AfterCount() trigger seems to be different between python sdk and the java one, so I created a few tests to show the difference, but in general I think the python sdk will buffer on result instead of input elements.
>> >> >> >> >
>> >> >> >> > What do you guys think?
>> >> >> >> >
>> >> >> >> > and here are the tests. I ran them in batch mode.
>> >> >> >> >
>> >> >> >> > Sincerely,
>> >> >> >> > Leiyi

Re: python AfterCount() trigger behavior not aligned with java

[Leiyi Zhang <le...@google.com>]

Thank you for explaining the details Robert!

I do have 1 more question: is there a way in beam that allows triggering
after n events arrived at the gbk step from previous parts of a pipeline?
because afterCount() trigger for gbk is per-key, then it cannot be used
here right?

On Mon, Aug 17, 2020 at 3:22 PM Robert Bradshaw <ro...@google.com> wrote:

> Yeah, this is another subtlety. There's a notion of "window garbage
> collection" that's distinct from "window closing." Garbage collection
> happens regardless of whether the trigger was set iff the window is
> non-empty when the watermark + allowed lateness exceeds the end of
> window. (Well, there's a flag to control this behavior.) This has
> changed over time and I think you've uncovered a bug that the two SDKs
> are not consistent here.
>
> On Mon, Aug 17, 2020 at 2:29 PM Leiyi Zhang <le...@google.com> wrote:
> >
> > is it just for the python sdk or for both python and java sdk?
> > seems like java sdk will output result even if there are less than 3
> elements per key.
> >
> > On Mon, Aug 17, 2020 at 2:20 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>
> >> Yes, GBK is non-determanistic in the face of triggers as well. All
> >> triggers are per-key, evaluated independently for each key, so it'd be
> >> "do I have at least 3 results for this key."
> >>
> >> On Mon, Aug 17, 2020 at 2:08 PM Leiyi Zhang <le...@google.com> wrote:
> >> >
> >> > Thank you very much for the reply,
> >> > Is the result of gbk non-deterministic as well? between "do I have at
> least 3 results PER KEY" vs "do I have at least 3 incoming events before I
> trigger GBK"
> >> >
> >> > On Mon, Aug 17, 2020 at 1:39 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >> >>
> >> >> Triggers in beam are non-determanistic; both behaviors are acceptable
> >> >> (especially for batch mode). In practice, production runners evaluate
> >> >> triggers (e.g. in this case "do I have at least two elements")
> whenver
> >> >> a new batch of data comes in (for the Python direct runner, in batch
> >> >> mode, all the data comes in at once). To have more control over this
> >> >> you can use TestPipeline, which will attempt to fire triggers as
> >> >> eagerly as possible.
> >> >>
> >> >> On Mon, Aug 17, 2020 at 1:16 PM Leiyi Zhang <le...@google.com>
> wrote:
> >> >> >
> >> >> > for GBK wtih AfterCount(3), java sdk results in this and python
> sdk results in this
> >> >> >
> >> >> > for global count with aftercount(2), java sdk results in 2 2 2 2
> and python sdk results in 8.
> >> >> >
> >> >> > On Mon, Aug 17, 2020 at 12:40 PM Robert Bradshaw <
> robertwb@google.com> wrote:
> >> >> >>
> >> >> >> What results do you get in each?
> >> >> >>
> >> >> >> On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <le...@google.com>
> wrote:
> >> >> >> >
> >> >> >> > Hi everyone!
> >> >> >> > I noticed that the behavior of AfterCount() trigger seems to be
> different between python sdk and the java one, so I created a few tests to
> show the difference, but in general I think the python sdk will buffer on
> result instead of input elements.
> >> >> >> >
> >> >> >> > What do you guys think?
> >> >> >> >
> >> >> >> > and here are the tests. I ran them in batch mode.
> >> >> >> >
> >> >> >> > Sincerely,
> >> >> >> > Leiyi
>

Re: python AfterCount() trigger behavior not aligned with java

[Robert Bradshaw <ro...@google.com>]

Yeah, this is another subtlety. There's a notion of "window garbage
collection" that's distinct from "window closing." Garbage collection
happens regardless of whether the trigger was set iff the window is
non-empty when the watermark + allowed lateness exceeds the end of
window. (Well, there's a flag to control this behavior.) This has
changed over time and I think you've uncovered a bug that the two SDKs
are not consistent here.

On Mon, Aug 17, 2020 at 2:29 PM Leiyi Zhang <le...@google.com> wrote:
>
> is it just for the python sdk or for both python and java sdk?
> seems like java sdk will output result even if there are less than 3 elements per key.
>
> On Mon, Aug 17, 2020 at 2:20 PM Robert Bradshaw <ro...@google.com> wrote:
>>
>> Yes, GBK is non-determanistic in the face of triggers as well. All
>> triggers are per-key, evaluated independently for each key, so it'd be
>> "do I have at least 3 results for this key."
>>
>> On Mon, Aug 17, 2020 at 2:08 PM Leiyi Zhang <le...@google.com> wrote:
>> >
>> > Thank you very much for the reply,
>> > Is the result of gbk non-deterministic as well? between "do I have at least 3 results PER KEY" vs "do I have at least 3 incoming events before I trigger GBK"
>> >
>> > On Mon, Aug 17, 2020 at 1:39 PM Robert Bradshaw <ro...@google.com> wrote:
>> >>
>> >> Triggers in beam are non-determanistic; both behaviors are acceptable
>> >> (especially for batch mode). In practice, production runners evaluate
>> >> triggers (e.g. in this case "do I have at least two elements") whenver
>> >> a new batch of data comes in (for the Python direct runner, in batch
>> >> mode, all the data comes in at once). To have more control over this
>> >> you can use TestPipeline, which will attempt to fire triggers as
>> >> eagerly as possible.
>> >>
>> >> On Mon, Aug 17, 2020 at 1:16 PM Leiyi Zhang <le...@google.com> wrote:
>> >> >
>> >> > for GBK wtih AfterCount(3), java sdk results in this and python sdk results in this
>> >> >
>> >> > for global count with aftercount(2), java sdk results in 2 2 2 2 and python sdk results in 8.
>> >> >
>> >> > On Mon, Aug 17, 2020 at 12:40 PM Robert Bradshaw <ro...@google.com> wrote:
>> >> >>
>> >> >> What results do you get in each?
>> >> >>
>> >> >> On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <le...@google.com> wrote:
>> >> >> >
>> >> >> > Hi everyone!
>> >> >> > I noticed that the behavior of AfterCount() trigger seems to be different between python sdk and the java one, so I created a few tests to show the difference, but in general I think the python sdk will buffer on result instead of input elements.
>> >> >> >
>> >> >> > What do you guys think?
>> >> >> >
>> >> >> > and here are the tests. I ran them in batch mode.
>> >> >> >
>> >> >> > Sincerely,
>> >> >> > Leiyi

Re: python AfterCount() trigger behavior not aligned with java

[Leiyi Zhang <le...@google.com>]

is it just for the python sdk or for both python and java sdk?
seems like java sdk will output result even if there are less than 3
elements per key.

On Mon, Aug 17, 2020 at 2:20 PM Robert Bradshaw <ro...@google.com> wrote:

> Yes, GBK is non-determanistic in the face of triggers as well. All
> triggers are per-key, evaluated independently for each key, so it'd be
> "do I have at least 3 results for this key."
>
> On Mon, Aug 17, 2020 at 2:08 PM Leiyi Zhang <le...@google.com> wrote:
> >
> > Thank you very much for the reply,
> > Is the result of gbk non-deterministic as well? between "do I have at
> least 3 results PER KEY" vs "do I have at least 3 incoming events before I
> trigger GBK"
> >
> > On Mon, Aug 17, 2020 at 1:39 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>
> >> Triggers in beam are non-determanistic; both behaviors are acceptable
> >> (especially for batch mode). In practice, production runners evaluate
> >> triggers (e.g. in this case "do I have at least two elements") whenver
> >> a new batch of data comes in (for the Python direct runner, in batch
> >> mode, all the data comes in at once). To have more control over this
> >> you can use TestPipeline, which will attempt to fire triggers as
> >> eagerly as possible.
> >>
> >> On Mon, Aug 17, 2020 at 1:16 PM Leiyi Zhang <le...@google.com> wrote:
> >> >
> >> > for GBK wtih AfterCount(3), java sdk results in this and python sdk
> results in this
> >> >
> >> > for global count with aftercount(2), java sdk results in 2 2 2 2 and
> python sdk results in 8.
> >> >
> >> > On Mon, Aug 17, 2020 at 12:40 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >> >>
> >> >> What results do you get in each?
> >> >>
> >> >> On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <le...@google.com>
> wrote:
> >> >> >
> >> >> > Hi everyone!
> >> >> > I noticed that the behavior of AfterCount() trigger seems to be
> different between python sdk and the java one, so I created a few tests to
> show the difference, but in general I think the python sdk will buffer on
> result instead of input elements.
> >> >> >
> >> >> > What do you guys think?
> >> >> >
> >> >> > and here are the tests. I ran them in batch mode.
> >> >> >
> >> >> > Sincerely,
> >> >> > Leiyi
>

Re: python AfterCount() trigger behavior not aligned with java

[Robert Bradshaw <ro...@google.com>]

Yes, GBK is non-determanistic in the face of triggers as well. All
triggers are per-key, evaluated independently for each key, so it'd be
"do I have at least 3 results for this key."

On Mon, Aug 17, 2020 at 2:08 PM Leiyi Zhang <le...@google.com> wrote:
>
> Thank you very much for the reply,
> Is the result of gbk non-deterministic as well? between "do I have at least 3 results PER KEY" vs "do I have at least 3 incoming events before I trigger GBK"
>
> On Mon, Aug 17, 2020 at 1:39 PM Robert Bradshaw <ro...@google.com> wrote:
>>
>> Triggers in beam are non-determanistic; both behaviors are acceptable
>> (especially for batch mode). In practice, production runners evaluate
>> triggers (e.g. in this case "do I have at least two elements") whenver
>> a new batch of data comes in (for the Python direct runner, in batch
>> mode, all the data comes in at once). To have more control over this
>> you can use TestPipeline, which will attempt to fire triggers as
>> eagerly as possible.
>>
>> On Mon, Aug 17, 2020 at 1:16 PM Leiyi Zhang <le...@google.com> wrote:
>> >
>> > for GBK wtih AfterCount(3), java sdk results in this and python sdk results in this
>> >
>> > for global count with aftercount(2), java sdk results in 2 2 2 2 and python sdk results in 8.
>> >
>> > On Mon, Aug 17, 2020 at 12:40 PM Robert Bradshaw <ro...@google.com> wrote:
>> >>
>> >> What results do you get in each?
>> >>
>> >> On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <le...@google.com> wrote:
>> >> >
>> >> > Hi everyone!
>> >> > I noticed that the behavior of AfterCount() trigger seems to be different between python sdk and the java one, so I created a few tests to show the difference, but in general I think the python sdk will buffer on result instead of input elements.
>> >> >
>> >> > What do you guys think?
>> >> >
>> >> > and here are the tests. I ran them in batch mode.
>> >> >
>> >> > Sincerely,
>> >> > Leiyi

Re: python AfterCount() trigger behavior not aligned with java

[Leiyi Zhang <le...@google.com>]

Thank you very much for the reply,
Is the result of gbk non-deterministic as well? between "do I have at least
3 results PER KEY" vs "do I have at least 3 incoming events before I
trigger GBK"

On Mon, Aug 17, 2020 at 1:39 PM Robert Bradshaw <ro...@google.com> wrote:

> Triggers in beam are non-determanistic; both behaviors are acceptable
> (especially for batch mode). In practice, production runners evaluate
> triggers (e.g. in this case "do I have at least two elements") whenver
> a new batch of data comes in (for the Python direct runner, in batch
> mode, all the data comes in at once). To have more control over this
> you can use TestPipeline, which will attempt to fire triggers as
> eagerly as possible.
>
> On Mon, Aug 17, 2020 at 1:16 PM Leiyi Zhang <le...@google.com> wrote:
> >
> > for GBK wtih AfterCount(3), java sdk results in this and python sdk
> results in this
> >
> > for global count with aftercount(2), java sdk results in 2 2 2 2 and
> python sdk results in 8.
> >
> > On Mon, Aug 17, 2020 at 12:40 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>
> >> What results do you get in each?
> >>
> >> On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <le...@google.com> wrote:
> >> >
> >> > Hi everyone!
> >> > I noticed that the behavior of AfterCount() trigger seems to be
> different between python sdk and the java one, so I created a few tests to
> show the difference, but in general I think the python sdk will buffer on
> result instead of input elements.
> >> >
> >> > What do you guys think?
> >> >
> >> > and here are the tests. I ran them in batch mode.
> >> >
> >> > Sincerely,
> >> > Leiyi
>

Re: python AfterCount() trigger behavior not aligned with java

[Robert Bradshaw <ro...@google.com>]

Triggers in beam are non-determanistic; both behaviors are acceptable
(especially for batch mode). In practice, production runners evaluate
triggers (e.g. in this case "do I have at least two elements") whenver
a new batch of data comes in (for the Python direct runner, in batch
mode, all the data comes in at once). To have more control over this
you can use TestPipeline, which will attempt to fire triggers as
eagerly as possible.

On Mon, Aug 17, 2020 at 1:16 PM Leiyi Zhang <le...@google.com> wrote:
>
> for GBK wtih AfterCount(3), java sdk results in this and python sdk results in this
>
> for global count with aftercount(2), java sdk results in 2 2 2 2 and python sdk results in 8.
>
> On Mon, Aug 17, 2020 at 12:40 PM Robert Bradshaw <ro...@google.com> wrote:
>>
>> What results do you get in each?
>>
>> On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <le...@google.com> wrote:
>> >
>> > Hi everyone!
>> > I noticed that the behavior of AfterCount() trigger seems to be different between python sdk and the java one, so I created a few tests to show the difference, but in general I think the python sdk will buffer on result instead of input elements.
>> >
>> > What do you guys think?
>> >
>> > and here are the tests. I ran them in batch mode.
>> >
>> > Sincerely,
>> > Leiyi

Re: python AfterCount() trigger behavior not aligned with java

[Leiyi Zhang <le...@google.com>]

for GBK wtih AfterCount(3), java sdk results in this
<https://screenshot.googleplex.com/a88897fd-f95b-468d-96a2-447db9b5effa> and
python sdk results in this
<https://screenshot.googleplex.com/6b0c5fb1-53be-4dac-9bd0-1221f8775e21>

for global count with aftercount(2), java sdk results in 2 2 2 2 and python
sdk results in 8.

On Mon, Aug 17, 2020 at 12:40 PM Robert Bradshaw <ro...@google.com>
wrote:

> What results do you get in each?
>
> On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <le...@google.com> wrote:
> >
> > Hi everyone!
> > I noticed that the behavior of AfterCount() trigger seems to be
> different between python sdk and the java one, so I created a few tests to
> show the difference, but in general I think the python sdk will buffer on
> result instead of input elements.
> >
> > What do you guys think?
> >
> > and here are the tests. I ran them in batch mode.
> >
> > Sincerely,
> > Leiyi
>

Re: python AfterCount() trigger behavior not aligned with java

[Robert Bradshaw <ro...@google.com>]

What results do you get in each?

On Mon, Aug 17, 2020 at 11:55 AM Leiyi Zhang <le...@google.com> wrote:
>
> Hi everyone!
> I noticed that the behavior of AfterCount() trigger seems to be different between python sdk and the java one, so I created a few tests to show the difference, but in general I think the python sdk will buffer on result instead of input elements.
>
> What do you guys think?
>
> and here are the tests. I ran them in batch mode.
>
> Sincerely,
> Leiyi