You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by "Plajt, Vaclav" <Va...@firma.seznam.cz> on 2018/05/10 13:38:09 UTC

triggers in direct runner

Hello Beam Devs,
I'm working on DSL-Euphoria. And I found that when GroupByKey transform is executed on direct runner, window triggers are evaluated element-wise (ReduceFnRunner#processElement) but not actually fired element-wise. They are fired (pane is emitted) when whole batch of elements is processed (ReduceFnRunner#processElements).

I also suspects that elements are already grouped by key when they reach ReduceFnRunner#processElements so triggers are fired only once per whole group.

I'm confused by this behavior since Beam Programming guide has examples with element-wise triggerning (see paragraph 8.4.1.1.).

Is this a wanted behavior? And does every Beam Runner do it the same (portability)? Or do I miss something?

Any kind of help or guidance is appreciated.
Thank you.
Vaclav Plajt


Je dobré vedet, ze tento e-mail a prílohy jsou duverné. Pokud spolu jednáme o uzavrení obchodu, vyhrazujeme si právo nase jednání kdykoli ukoncit. Pro fanousky právní mluvy - vylucujeme tím ustanovení obcanského zákoníku o predsmluvní odpovednosti. Pravidla o tom, kdo u nás a jak vystupuje za spolecnost a kdo muze co a jak podepsat naleznete zde<https://onas.seznam.cz/cz/podpisovy-rad-cz.html>

You should know that this e-mail and its attachments are confidential. If we are negotiating on the conclusion of a transaction, we reserve the right to terminate the negotiations at any time. For fans of legalese-we hereby exclude the provisions of the Civil Code on pre-contractual liability. The rules about who and how may act for the company and what are the signing procedures can be found here<https://onas.seznam.cz/cz/signature-rules.html>.

Re: triggers in direct runner

Posted by "Plajt, Vaclav" <Va...@firma.seznam.cz>.
Hi Kenneth,

thanks for the clarification. I was not aware of bundles. Now it makes sense.


Vaclav

________________________________
From: Kenneth Knowles <kl...@google.com>
Sent: Thursday, May 10, 2018 4:57:34 PM
To: dev
Subject: Re: triggers in direct runner

Hi Vaclav,

Slightly stale but still maybe a good reference is https://s.apache.org/beam-triggers.

Triggers are per key and window, and they give the runner permission to fire but do not require it. The runner can thus amortize the cost of output. The bundle is the unit of commit in Beam so firing sub-bundle is wasted.

Also relevant is https://s.apache.org/beam-sink-triggers. It emphasizes the other point: triggering is about completeness, latency, and cost. It is (mostly) a bug that triggering affects the observable elements.

Kenn


On Thu, May 10, 2018, 06:38 Plajt, Vaclav <Va...@firma.seznam.cz>> wrote:
Hello Beam Devs,
I'm working on DSL-Euphoria. And I found that when GroupByKey transform is executed on direct runner, window triggers are evaluated element-wise (ReduceFnRunner#processElement) but not actually fired element-wise. They are fired (pane is emitted) when whole batch of elements is processed (ReduceFnRunner#processElements).

I also suspects that elements are already grouped by key when they reach ReduceFnRunner#processElements so triggers are fired only once per whole group.

I'm confused by this behavior since Beam Programming guide has examples with element-wise triggerning (see paragraph 8.4.1.1.).

Is this a wanted behavior? And does every Beam Runner do it the same (portability)? Or do I miss something?

Any kind of help or guidance is appreciated.
Thank you.
Vaclav Plajt


Je dobré vědět, že tento e-mail a přílohy jsou důvěrné. Pokud spolu jednáme o uzavření obchodu, vyhrazujeme si právo naše jednání kdykoli ukončit. Pro fanoušky právní mluvy - vylučujeme tím ustanovení občanského zákoníku o předsmluvní odpovědnosti. Pravidla o tom, kdo u nás a jak vystupuje za společnost a kdo může co a jak podepsat naleznete zde<https://onas.seznam.cz/cz/podpisovy-rad-cz.html>

You should know that this e-mail and its attachments are confidential. If we are negotiating on the conclusion of a transaction, we reserve the right to terminate the negotiations at any time. For fans of legalese—we hereby exclude the provisions of the Civil Code on pre-contractual liability. The rules about who and how may act for the company and what are the signing procedures can be found here<https://onas.seznam.cz/cz/signature-rules.html>.

Re: triggers in direct runner

Posted by Kenneth Knowles <kl...@google.com>.
Hi Vaclav,

Slightly stale but still maybe a good reference is
https://s.apache.org/beam-triggers.

Triggers are per key and window, and they give the runner permission to
fire but do not require it. The runner can thus amortize the cost of
output. The bundle is the unit of commit in Beam so firing sub-bundle is
wasted.

Also relevant is https://s.apache.org/beam-sink-triggers. It emphasizes the
other point: triggering is about completeness, latency, and cost. It is
(mostly) a bug that triggering affects the observable elements.

Kenn


On Thu, May 10, 2018, 06:38 Plajt, Vaclav <Va...@firma.seznam.cz>
wrote:

> Hello Beam Devs,
> I'm working on DSL-Euphoria. And I found that when GroupByKey transform is
> executed on direct runner, window triggers are evaluated element-wise
> (ReduceFnRunner#processElement) but not actually fired element-wise. They
> are fired (pane is emitted) when whole batch of elements is processed
> (ReduceFnRunner#processElements).
>
> I also suspects that elements are already grouped by key when they reach
> ReduceFnRunner#processElements so triggers are fired only once per whole
> group.
>
> I'm confused by this behavior since Beam Programming guide has examples
> with element-wise triggerning (see paragraph 8.4.1.1.).
>
> Is this a wanted behavior? And does every Beam Runner do it the same
> (portability)? Or do I miss something?
>
> Any kind of help or guidance is appreciated.
> Thank you.
> Vaclav Plajt
>
>
> Je dobré vědět, že tento e-mail a přílohy jsou důvěrné. Pokud spolu
> jednáme o uzavření obchodu, vyhrazujeme si právo naše jednání kdykoli
> ukončit. Pro fanoušky právní mluvy - vylučujeme tím ustanovení občanského
> zákoníku o předsmluvní odpovědnosti. Pravidla o tom, kdo u nás a jak
> vystupuje za společnost a kdo může co a jak podepsat naleznete zde
> <https://onas.seznam.cz/cz/podpisovy-rad-cz.html>
>
> You should know that this e-mail and its attachments are confidential. If
> we are negotiating on the conclusion of a transaction, we reserve the right
> to terminate the negotiations at any time. For fans of legalese—we hereby
> exclude the provisions of the Civil Code on pre-contractual liability. The
> rules about who and how may act for the company and what are the signing
> procedures can be found here
> <https://onas.seznam.cz/cz/signature-rules.html>.
>