You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Kenneth Knowles <kl...@google.com> on 2017/11/30 20:06:41 UTC
[DISCUSS] [DOC] Triggering is for sinks!
Hi all,
Triggers are one of the more novel aspects of Beam's support for unbounded
data. They are also one of the most challenging aspects of the model.
Ben & I have been working on a major new idea for how triggers could work
in the Beam model. We think it will make triggers much more usable, create
new opportunities for no-knobs execution/optimization, and improve
compatibility with DSLs like SQL. (also eliminate a whole class of bugs)
Triggering is for sinks!
https://s.apache.org/beam-sink-triggers
Please take a look at this "1"-pager and give feedback.
Kenn
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Frances Perry <fr...@apache.org>.
I love this idea -- long live the separation of business logic and runtime
requirements.
On Thu, Nov 30, 2017 at 12:06 PM, Kenneth Knowles <kl...@google.com> wrote:
> Hi all,
>
> Triggers are one of the more novel aspects of Beam's support for unbounded
> data. They are also one of the most challenging aspects of the model.
>
> Ben & I have been working on a major new idea for how triggers could work
> in the Beam model. We think it will make triggers much more usable, create
> new opportunities for no-knobs execution/optimization, and improve
> compatibility with DSLs like SQL. (also eliminate a whole class of bugs)
>
> Triggering is for sinks!
>
> https://s.apache.org/beam-sink-triggers
>
> Please take a look at this "1"-pager and give feedback.
>
> Kenn
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Dan Sotolongo <ds...@google.com>.
On Tue, Dec 5, 2017 at 12:00 PM, Kenneth Knowles <kl...@google.com> wrote:
> in the unbounded case I would like to be specified as having to agree with
> the bounded spec for any finite prefix. I'm not sure if an operational view
> is amenable to this.
>
It seems to me that the only way to reconcile the two views is with perfect
watermarks. Historically, we've taken the position that watermarks are
necessarily heuristic, but a change in perspective could yield gains in
model simplicity. In particular, thinking of late data as an error and late
triggers as error-handling code (which would typically just lead to an
alert) would simplify pipeline logic significantly in cases where what to
do with late data is non-obvious. At present, watermark-correctness is
something that's not well-supported and over which we don't have much
control, but the situation will likely improve as consistent streaming
gains market share. Reconsidering the merits of heuristic vs perfect
watermarks would be a natural thing to do as that happens.
and then there's merging...
>
FWIW, your argument for windowing being an important property of a
PCollection still holds for merging windows. It's just more complex because
the analogy between windows and keys breaks down. The windows remain a
property of the elements within the PCollection that affects the shape of
any "tables" that will be computed with it.
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Kenneth Knowles <kl...@google.com>.
On Wed, Dec 6, 2017 at 10:06 PM, Reuven Lax <re...@google.com> wrote:
>
>
>>> What are the instances where windowing is used without an aggregation?
>>>
>>
>> One example is in our destination sinks. Mapping to a destination is
>> usually done by simply examining the window on an element, and these sinks
>> generally do not group by that window.
>>
>
But a KV (or nested as needed) would suffice for this, yes? Seems like the
window is a non-special piece of data.
Kenn
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Reuven Lax <re...@google.com>.
That should have read our dynamic sinks.
On Wed, Dec 6, 2017 at 10:05 PM, Reuven Lax <re...@google.com> wrote:
>
>
> On Wed, Dec 6, 2017 at 9:53 PM, Kenneth Knowles <kl...@google.com> wrote:
>
>>
>>
>> On Wed, Dec 6, 2017 at 9:45 PM, Reuven Lax <re...@google.com> wrote:
>>>
>>> Ignoring merging, one perspective is that the window is just a key with
>>>>>> a deadline.
>>>>>>
>>>>>
>>>>> That is only true when performing an aggregation. Records can be
>>>>> associated with a window, and do not require keys at that point. The
>>>>> "deadline" only applies when something like a GBK is assigned.
>>>>>
>>>>
>>>> Yea, that situation -- windows assigned but no aggregation yet -- is
>>>> analogous to data being a KV prior to the GBK. The main function that
>>>> windows actually serve in the life of data processing is to allow
>>>> aggregations over unbounded data with bounded resources. Only aggregation
>>>> really needs them - if you just have a pass-through sequence of ParDos
>>>> windows don't really do anything.
>>>>
>>>
>>> I disagree. There are multiple instances where windowing is used without
>>> an aggregation after. Fundamentally windowing is a function on elements.
>>> This function is used during aggregations to bound aggregations, but makes
>>> sense on its own. Thinking of windowing as a "timeout" makes for an
>>> intuitive model, but I don't think it's really the right model. For one
>>> thing, that intuitive model makes less sense in batch.
>>>
>>
>> What are the instances where windowing is used without an aggregation?
>>
>
> One example is in our destination sinks. Mapping to a destination is
> usually done by simply examining the window on an element, and these sinks
> generally do not group by that window.
>
>
>> Kenn
>>
>>
>>
>>
>>>
>>>
>>>
>>>> Kenn
>>>>
>>>>
>>>>> From this perspective, the distinction between key and window is not
>>>>>> important; you could just say that GBK requires the composite key for a
>>>>>> group to eventually expire (in SQL terms, you just need one of the GROUP BY
>>>>>> arguments to provide the deadline, and they are otherwise all on equal
>>>>>> footing). And so the window is just as much a part of the data as the key.
>>>>>> Without merging, once it is assigned you don't need to keep around the
>>>>>> WindowFn or any such. Of course, our way of automatically propagating
>>>>>> windows from inputs to outputs, akin to making MapValues the default mode
>>>>>> of computation, requires the window to be a distinguished secondary key.
>>>>>>
>>>>>> Another way I think about it is that the windowing + watermark +
>>>>>> allowed lateness defines which elements are a part of a PCollection and
>>>>>> which are not. Dropped data semantically never existed in the first place.
>>>>>> This was actually independent of windowing before the "window expiration"
>>>>>> model of dropping data. I still think window expiration + GC + dropping go
>>>>>> together nicely, and drop less data needlessly, but just dropping data
>>>>>> behind the watermark + allowed lateness has some appeal for isolating the
>>>>>> operational aspect here.
>>>>>>
>>>>>> Operationally, you might take the view that the act of expiration and
>>>>>> dropping all remaining data is a configuration on the GBK. Then the
>>>>>> WindowingStrategy, like windows and KV, are plumbing devices to reach a GBK
>>>>>> that may be deep in a composite (which is certainly true anyhow). I don't
>>>>>> really like this, because I would like the output of a GBK to be a
>>>>>> straightforward function of its input - in the unbounded case I would like
>>>>>> to be specified as having to agree with the bounded spec for any finite
>>>>>> prefix. I'm not sure if an operational view is amenable to this. If they
>>>>>> both work, then being able to switch perspectives back and forth would be
>>>>>> cool.
>>>>>>
>>>>>> I think there are some inconsistencies in the above intuitions, and
>>>>>> then there's merging...
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>>
>>>>>> Also, I think anyone reading this document really ought to at least
>>>>>>> skim the (linked from there) http://s.apache.org/beam-streams-tables and
>>>>>>> internalize the idea of "PCollections as changelogs, aggregations as tables
>>>>>>> on which the changelog acts". It probably would be good to rewrite our
>>>>>>> documentation with this in mind: even with my experience on the Beam team,
>>>>>>> this simple idea made it much easier for me to think clearly about all the
>>>>>>> concepts.
>>>>>>>
>>>>>>> I'm very excited about both of these ideas, I think they rival in
>>>>>>> importance the idea of batch/streaming unification and will end up being a
>>>>>>> fundamental part of the future of Beam model.
>>>>>>>
>>>>>>> On Thu, Nov 30, 2017 at 8:52 PM Jean-Baptiste Onofré <
>>>>>>> jb@nanthrax.net> wrote:
>>>>>>>
>>>>>>>> Hi Kenn,
>>>>>>>>
>>>>>>>> very interesting idea. It sounds more usable and "logic".
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> JB
>>>>>>>>
>>>>>>>> On 11/30/2017 09:06 PM, Kenneth Knowles wrote:
>>>>>>>> > Hi all,
>>>>>>>> >
>>>>>>>> > Triggers are one of the more novel aspects of Beam's support for
>>>>>>>> unbounded data.
>>>>>>>> > They are also one of the most challenging aspects of the model.
>>>>>>>> >
>>>>>>>> > Ben & I have been working on a major new idea for how triggers
>>>>>>>> could work in the
>>>>>>>> > Beam model. We think it will make triggers much more usable,
>>>>>>>> create new
>>>>>>>> > opportunities for no-knobs execution/optimization, and improve
>>>>>>>> compatibility
>>>>>>>> > with DSLs like SQL. (also eliminate a whole class of bugs)
>>>>>>>> >
>>>>>>>> > Triggering is for sinks!
>>>>>>>> >
>>>>>>>> > https://s.apache.org/beam-sink-triggers
>>>>>>>> >
>>>>>>>> > Please take a look at this "1"-pager and give feedback.
>>>>>>>> >
>>>>>>>> > Kenn
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jean-Baptiste Onofré
>>>>>>>> jbonofre@apache.org
>>>>>>>> http://blog.nanthrax.net
>>>>>>>> Talend - http://www.talend.com
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Reuven Lax <re...@google.com>.
On Wed, Dec 6, 2017 at 9:53 PM, Kenneth Knowles <kl...@google.com> wrote:
>
>
> On Wed, Dec 6, 2017 at 9:45 PM, Reuven Lax <re...@google.com> wrote:
>>
>> Ignoring merging, one perspective is that the window is just a key with a
>>>>> deadline.
>>>>>
>>>>
>>>> That is only true when performing an aggregation. Records can be
>>>> associated with a window, and do not require keys at that point. The
>>>> "deadline" only applies when something like a GBK is assigned.
>>>>
>>>
>>> Yea, that situation -- windows assigned but no aggregation yet -- is
>>> analogous to data being a KV prior to the GBK. The main function that
>>> windows actually serve in the life of data processing is to allow
>>> aggregations over unbounded data with bounded resources. Only aggregation
>>> really needs them - if you just have a pass-through sequence of ParDos
>>> windows don't really do anything.
>>>
>>
>> I disagree. There are multiple instances where windowing is used without
>> an aggregation after. Fundamentally windowing is a function on elements.
>> This function is used during aggregations to bound aggregations, but makes
>> sense on its own. Thinking of windowing as a "timeout" makes for an
>> intuitive model, but I don't think it's really the right model. For one
>> thing, that intuitive model makes less sense in batch.
>>
>
> What are the instances where windowing is used without an aggregation?
>
One example is in our destination sinks. Mapping to a destination is
usually done by simply examining the window on an element, and these sinks
generally do not group by that window.
> Kenn
>
>
>
>
>>
>>
>>
>>> Kenn
>>>
>>>
>>>> From this perspective, the distinction between key and window is not
>>>>> important; you could just say that GBK requires the composite key for a
>>>>> group to eventually expire (in SQL terms, you just need one of the GROUP BY
>>>>> arguments to provide the deadline, and they are otherwise all on equal
>>>>> footing). And so the window is just as much a part of the data as the key.
>>>>> Without merging, once it is assigned you don't need to keep around the
>>>>> WindowFn or any such. Of course, our way of automatically propagating
>>>>> windows from inputs to outputs, akin to making MapValues the default mode
>>>>> of computation, requires the window to be a distinguished secondary key.
>>>>>
>>>>> Another way I think about it is that the windowing + watermark +
>>>>> allowed lateness defines which elements are a part of a PCollection and
>>>>> which are not. Dropped data semantically never existed in the first place.
>>>>> This was actually independent of windowing before the "window expiration"
>>>>> model of dropping data. I still think window expiration + GC + dropping go
>>>>> together nicely, and drop less data needlessly, but just dropping data
>>>>> behind the watermark + allowed lateness has some appeal for isolating the
>>>>> operational aspect here.
>>>>>
>>>>> Operationally, you might take the view that the act of expiration and
>>>>> dropping all remaining data is a configuration on the GBK. Then the
>>>>> WindowingStrategy, like windows and KV, are plumbing devices to reach a GBK
>>>>> that may be deep in a composite (which is certainly true anyhow). I don't
>>>>> really like this, because I would like the output of a GBK to be a
>>>>> straightforward function of its input - in the unbounded case I would like
>>>>> to be specified as having to agree with the bounded spec for any finite
>>>>> prefix. I'm not sure if an operational view is amenable to this. If they
>>>>> both work, then being able to switch perspectives back and forth would be
>>>>> cool.
>>>>>
>>>>> I think there are some inconsistencies in the above intuitions, and
>>>>> then there's merging...
>>>>>
>>>>> Kenn
>>>>>
>>>>>
>>>>> Also, I think anyone reading this document really ought to at least
>>>>>> skim the (linked from there) http://s.apache.org/beam-streams-tables and
>>>>>> internalize the idea of "PCollections as changelogs, aggregations as tables
>>>>>> on which the changelog acts". It probably would be good to rewrite our
>>>>>> documentation with this in mind: even with my experience on the Beam team,
>>>>>> this simple idea made it much easier for me to think clearly about all the
>>>>>> concepts.
>>>>>>
>>>>>> I'm very excited about both of these ideas, I think they rival in
>>>>>> importance the idea of batch/streaming unification and will end up being a
>>>>>> fundamental part of the future of Beam model.
>>>>>>
>>>>>> On Thu, Nov 30, 2017 at 8:52 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Kenn,
>>>>>>>
>>>>>>> very interesting idea. It sounds more usable and "logic".
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On 11/30/2017 09:06 PM, Kenneth Knowles wrote:
>>>>>>> > Hi all,
>>>>>>> >
>>>>>>> > Triggers are one of the more novel aspects of Beam's support for
>>>>>>> unbounded data.
>>>>>>> > They are also one of the most challenging aspects of the model.
>>>>>>> >
>>>>>>> > Ben & I have been working on a major new idea for how triggers
>>>>>>> could work in the
>>>>>>> > Beam model. We think it will make triggers much more usable,
>>>>>>> create new
>>>>>>> > opportunities for no-knobs execution/optimization, and improve
>>>>>>> compatibility
>>>>>>> > with DSLs like SQL. (also eliminate a whole class of bugs)
>>>>>>> >
>>>>>>> > Triggering is for sinks!
>>>>>>> >
>>>>>>> > https://s.apache.org/beam-sink-triggers
>>>>>>> >
>>>>>>> > Please take a look at this "1"-pager and give feedback.
>>>>>>> >
>>>>>>> > Kenn
>>>>>>>
>>>>>>> --
>>>>>>> Jean-Baptiste Onofré
>>>>>>> jbonofre@apache.org
>>>>>>> http://blog.nanthrax.net
>>>>>>> Talend - http://www.talend.com
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Robert Bradshaw <ro...@google.com>.
+1
Windowing is not operational, and in fact doesn't rely on the notions
of watermarks at all. Watermarks are an operational tool used to
compute windowing for streaming pipelines, and failure to respect the
perfect watermark results in a failure to correctly compute windowing.
We then thrown the operational notion of triggering to give pipeline
authors the ability to control and try to deal with these failures.
On Thu, Dec 7, 2017 at 8:53 AM, Reuven Lax <re...@google.com> wrote:
> My point is that windowing in the abstract is not fundamentally
> operational, as proven by the fact that in batch pipelines windowing is
> perfectly determined by the data. In streaming pipelines things turn a bit
> more operational, but that's not fundamental to the data model. In fact, you
> can model the watermark as a perfect magical oracle (you can just define it
> as a function from a prefix of the stream to a timestamp, not involving
> processing time at all); this isn't true in real life of course, but it does
> allow the watermark to be modeled in a non-operational way.
>
> On Thu, Dec 7, 2017 at 1:02 AM, Kenneth Knowles <kl...@google.com> wrote:
>>
>> On Wed, Dec 6, 2017 at 10:44 PM, Robert Bradshaw <ro...@google.com>
>> wrote:
>>>
>>> So, to answer the original question, I don't think windowing is
>>> operational, as given a particular windowing there is one and only one
>>> right output for a given input.
>>
>>
>> I just want to highlight that this is an operational/semantic blend. The
>> watermark determines what is in or is not in a PCollection, and the
>> watermark is fundamentally operational, since it is a relation between event
>> time and processing time. I am in complete agreement with your statement
>> _for a particular watermark curve_.
>>
>> Or, more aggressively, if we say that dropped elements never existed - by
>> definition, and according to the watermark curve - then every window is
>> bounded (or the pipeline is stuck) and there is one and only one right
>> output and it will match a batch job over the same.
>>
>> This touches on what I mean by the simplicity of dropping data based on
>> (watermark - timestamp) instead of (watermark - window expiry). When
>> dropping is based on window expiry, the relationship between real-time
>> results and archival results is not as clear.
>>
>> Kenn
>>
>>
>>>
>>> This doesn't mean that it couldn't be
>>> interesting to explore expressing the desired windowing on the output
>>> of a transformation and letting it flow up rather than expressing it
>>> on the input and letting it flow down. (There are technical
>>> implications here--an output consumed in multiple ways may need to be
>>> executed multiple times despite it occurring once in the graph which
>>> could be counter-intuitive, though even just changing the triggering
>>> could require this, and runners could probably often be intelligent
>>> about sharing work between similar-enough windowing. It also means the
>>> choice of windowing couldn't be inspected during construction (though
>>> again we may have crossed that bridge if we can't inspect the choice
>>> of triggering).)
>>>
>>> Triggering, on the other hand, is very operational, and intention more
>>> easily defined at a sink (and propagated upwards) rather than the way
>>> we do now, so a huge +1 to this proposal.
>>>
>>> - Robert
>>
>>
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Reuven Lax <re...@google.com>.
My point is that windowing in the abstract is not fundamentally
operational, as proven by the fact that in batch pipelines windowing is
perfectly determined by the data. In streaming pipelines things turn a bit
more operational, but that's not fundamental to the data model. In fact,
you can model the watermark as a perfect magical oracle (you can just
define it as a function from a prefix of the stream to a timestamp, not
involving processing time at all); this isn't true in real life of course,
but it does allow the watermark to be modeled in a non-operational way.
On Thu, Dec 7, 2017 at 1:02 AM, Kenneth Knowles <kl...@google.com> wrote:
> On Wed, Dec 6, 2017 at 10:44 PM, Robert Bradshaw <ro...@google.com>
> wrote:
>>
>> So, to answer the original question, I don't think windowing is
>> operational, as given a particular windowing there is one and only one
>> right output for a given input.
>
>
> I just want to highlight that this is an operational/semantic blend. The
> watermark determines what is in or is not in a PCollection, and the
> watermark is fundamentally operational, since it is a relation between
> event time and processing time. I am in complete agreement with your
> statement _for a particular watermark curve_.
>
> Or, more aggressively, if we say that dropped elements never existed - by
> definition, and according to the watermark curve - then every window is
> bounded (or the pipeline is stuck) and there is one and only one right
> output and it will match a batch job over the same.
>
> This touches on what I mean by the simplicity of dropping data based on
> (watermark - timestamp) instead of (watermark - window expiry). When
> dropping is based on window expiry, the relationship between real-time
> results and archival results is not as clear.
>
> Kenn
>
>
>
>> This doesn't mean that it couldn't be
>> interesting to explore expressing the desired windowing on the output
>> of a transformation and letting it flow up rather than expressing it
>> on the input and letting it flow down. (There are technical
>> implications here--an output consumed in multiple ways may need to be
>> executed multiple times despite it occurring once in the graph which
>> could be counter-intuitive, though even just changing the triggering
>> could require this, and runners could probably often be intelligent
>> about sharing work between similar-enough windowing. It also means the
>> choice of windowing couldn't be inspected during construction (though
>> again we may have crossed that bridge if we can't inspect the choice
>> of triggering).)
>>
>> Triggering, on the other hand, is very operational, and intention more
>> easily defined at a sink (and propagated upwards) rather than the way
>> we do now, so a huge +1 to this proposal.
>>
>> - Robert
>>
>
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Kenneth Knowles <kl...@google.com>.
On Wed, Dec 6, 2017 at 10:44 PM, Robert Bradshaw <ro...@google.com>
wrote:
>
> So, to answer the original question, I don't think windowing is
> operational, as given a particular windowing there is one and only one
> right output for a given input.
I just want to highlight that this is an operational/semantic blend. The
watermark determines what is in or is not in a PCollection, and the
watermark is fundamentally operational, since it is a relation between
event time and processing time. I am in complete agreement with your
statement _for a particular watermark curve_.
Or, more aggressively, if we say that dropped elements never existed - by
definition, and according to the watermark curve - then every window is
bounded (or the pipeline is stuck) and there is one and only one right
output and it will match a batch job over the same.
This touches on what I mean by the simplicity of dropping data based on
(watermark - timestamp) instead of (watermark - window expiry). When
dropping is based on window expiry, the relationship between real-time
results and archival results is not as clear.
Kenn
> This doesn't mean that it couldn't be
> interesting to explore expressing the desired windowing on the output
> of a transformation and letting it flow up rather than expressing it
> on the input and letting it flow down. (There are technical
> implications here--an output consumed in multiple ways may need to be
> executed multiple times despite it occurring once in the graph which
> could be counter-intuitive, though even just changing the triggering
> could require this, and runners could probably often be intelligent
> about sharing work between similar-enough windowing. It also means the
> choice of windowing couldn't be inspected during construction (though
> again we may have crossed that bridge if we can't inspect the choice
> of triggering).)
>
> Triggering, on the other hand, is very operational, and intention more
> easily defined at a sink (and propagated upwards) rather than the way
> we do now, so a huge +1 to this proposal.
>
> - Robert
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Robert Bradshaw <ro...@google.com>.
On Wed, Dec 6, 2017 at 9:53 PM, Kenneth Knowles <kl...@google.com> wrote:
>
> On Wed, Dec 6, 2017 at 9:45 PM, Reuven Lax <re...@google.com> wrote:
>>>>>
>>>>> Ignoring merging, one perspective is that the window is just a key with
>>>>> a deadline.
>>>>
>>>>
>>>> That is only true when performing an aggregation. Records can be
>>>> associated with a window, and do not require keys at that point. The
>>>> "deadline" only applies when something like a GBK is assigned.
>>>
>>>
>>> Yea, that situation -- windows assigned but no aggregation yet -- is
>>> analogous to data being a KV prior to the GBK. The main function that
>>> windows actually serve in the life of data processing is to allow
>>> aggregations over unbounded data with bounded resources. Only aggregation
>>> really needs them - if you just have a pass-through sequence of ParDos
>>> windows don't really do anything.
>>
>>
>> I disagree. There are multiple instances where windowing is used without
>> an aggregation after. Fundamentally windowing is a function on elements.
>> This function is used during aggregations to bound aggregations, but makes
>> sense on its own. Thinking of windowing as a "timeout" makes for an
>> intuitive model, but I don't think it's really the right model. For one
>> thing, that intuitive model makes less sense in batch.
>
> What are the instances where windowing is used without an aggregation?
There are cases where windowing is used without a group-by-key, e.g.
for state and side inputs, but to me these still are a kind of
aggregation, or more generally grouping of multiple distinct elements
into one.
My model of windows is that they form a secondary key. This key, along
with the primary key, is used in grouping (including the non-gbk
operations listed above), and grouping across these composite key is
not allowed in the model. (Merging windows is simply a case where the
actual key is not computed until the grouping is performed, but if we
had an oracle that could tell us, for example, which session an
element belonged to up front, then we could simply inject this into
the key explicitly and do a "normal" group-by-key. A corollary of this
means that acting on the (proto)window of a merging window fn before
aggregation happens should be disallowed, maybe there's a JIRA to file
here?)
The other novel properties associated with this time-based key are
that they have a known upper bound (so we can use watermarks to make
concrete progress before seeing all the data, though watermarks
themselves are not part of the model (at least not until triggers are
introduced)) and also that they are imposed externally and passed
explicitly.
So, to answer the original question, I don't think windowing is
operational, as given a particular windowing there is one and only one
right output for a given input. This doesn't mean that it couldn't be
interesting to explore expressing the desired windowing on the output
of a transformation and letting it flow up rather than expressing it
on the input and letting it flow down. (There are technical
implications here--an output consumed in multiple ways may need to be
executed multiple times despite it occurring once in the graph which
could be counter-intuitive, though even just changing the triggering
could require this, and runners could probably often be intelligent
about sharing work between similar-enough windowing. It also means the
choice of windowing couldn't be inspected during construction (though
again we may have crossed that bridge if we can't inspect the choice
of triggering).)
Triggering, on the other hand, is very operational, and intention more
easily defined at a sink (and propagated upwards) rather than the way
we do now, so a huge +1 to this proposal.
- Robert
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Kenneth Knowles <kl...@google.com>.
On Wed, Dec 6, 2017 at 9:45 PM, Reuven Lax <re...@google.com> wrote:
>
> Ignoring merging, one perspective is that the window is just a key with a
>>>> deadline.
>>>>
>>>
>>> That is only true when performing an aggregation. Records can be
>>> associated with a window, and do not require keys at that point. The
>>> "deadline" only applies when something like a GBK is assigned.
>>>
>>
>> Yea, that situation -- windows assigned but no aggregation yet -- is
>> analogous to data being a KV prior to the GBK. The main function that
>> windows actually serve in the life of data processing is to allow
>> aggregations over unbounded data with bounded resources. Only aggregation
>> really needs them - if you just have a pass-through sequence of ParDos
>> windows don't really do anything.
>>
>
> I disagree. There are multiple instances where windowing is used without
> an aggregation after. Fundamentally windowing is a function on elements.
> This function is used during aggregations to bound aggregations, but makes
> sense on its own. Thinking of windowing as a "timeout" makes for an
> intuitive model, but I don't think it's really the right model. For one
> thing, that intuitive model makes less sense in batch.
>
What are the instances where windowing is used without an aggregation?
Kenn
>
>
>
>> Kenn
>>
>>
>>> From this perspective, the distinction between key and window is not
>>>> important; you could just say that GBK requires the composite key for a
>>>> group to eventually expire (in SQL terms, you just need one of the GROUP BY
>>>> arguments to provide the deadline, and they are otherwise all on equal
>>>> footing). And so the window is just as much a part of the data as the key.
>>>> Without merging, once it is assigned you don't need to keep around the
>>>> WindowFn or any such. Of course, our way of automatically propagating
>>>> windows from inputs to outputs, akin to making MapValues the default mode
>>>> of computation, requires the window to be a distinguished secondary key.
>>>>
>>>> Another way I think about it is that the windowing + watermark +
>>>> allowed lateness defines which elements are a part of a PCollection and
>>>> which are not. Dropped data semantically never existed in the first place.
>>>> This was actually independent of windowing before the "window expiration"
>>>> model of dropping data. I still think window expiration + GC + dropping go
>>>> together nicely, and drop less data needlessly, but just dropping data
>>>> behind the watermark + allowed lateness has some appeal for isolating the
>>>> operational aspect here.
>>>>
>>>> Operationally, you might take the view that the act of expiration and
>>>> dropping all remaining data is a configuration on the GBK. Then the
>>>> WindowingStrategy, like windows and KV, are plumbing devices to reach a GBK
>>>> that may be deep in a composite (which is certainly true anyhow). I don't
>>>> really like this, because I would like the output of a GBK to be a
>>>> straightforward function of its input - in the unbounded case I would like
>>>> to be specified as having to agree with the bounded spec for any finite
>>>> prefix. I'm not sure if an operational view is amenable to this. If they
>>>> both work, then being able to switch perspectives back and forth would be
>>>> cool.
>>>>
>>>> I think there are some inconsistencies in the above intuitions, and
>>>> then there's merging...
>>>>
>>>> Kenn
>>>>
>>>>
>>>> Also, I think anyone reading this document really ought to at least
>>>>> skim the (linked from there) http://s.apache.org/beam-streams-tables and
>>>>> internalize the idea of "PCollections as changelogs, aggregations as tables
>>>>> on which the changelog acts". It probably would be good to rewrite our
>>>>> documentation with this in mind: even with my experience on the Beam team,
>>>>> this simple idea made it much easier for me to think clearly about all the
>>>>> concepts.
>>>>>
>>>>> I'm very excited about both of these ideas, I think they rival in
>>>>> importance the idea of batch/streaming unification and will end up being a
>>>>> fundamental part of the future of Beam model.
>>>>>
>>>>> On Thu, Nov 30, 2017 at 8:52 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>> wrote:
>>>>>
>>>>>> Hi Kenn,
>>>>>>
>>>>>> very interesting idea. It sounds more usable and "logic".
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 11/30/2017 09:06 PM, Kenneth Knowles wrote:
>>>>>> > Hi all,
>>>>>> >
>>>>>> > Triggers are one of the more novel aspects of Beam's support for
>>>>>> unbounded data.
>>>>>> > They are also one of the most challenging aspects of the model.
>>>>>> >
>>>>>> > Ben & I have been working on a major new idea for how triggers
>>>>>> could work in the
>>>>>> > Beam model. We think it will make triggers much more usable, create
>>>>>> new
>>>>>> > opportunities for no-knobs execution/optimization, and improve
>>>>>> compatibility
>>>>>> > with DSLs like SQL. (also eliminate a whole class of bugs)
>>>>>> >
>>>>>> > Triggering is for sinks!
>>>>>> >
>>>>>> > https://s.apache.org/beam-sink-triggers
>>>>>> >
>>>>>> > Please take a look at this "1"-pager and give feedback.
>>>>>> >
>>>>>> > Kenn
>>>>>>
>>>>>> --
>>>>>> Jean-Baptiste Onofré
>>>>>> jbonofre@apache.org
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Reuven Lax <re...@google.com>.
On Tue, Dec 5, 2017 at 2:16 PM, Kenneth Knowles <kl...@google.com> wrote:
>
>
> On Tue, Dec 5, 2017 at 1:50 PM, Reuven Lax <re...@google.com> wrote:
>
>>
>>
>> On Tue, Dec 5, 2017 at 12:00 PM, Kenneth Knowles <kl...@google.com> wrote:
>>
>>> On Tue, Dec 5, 2017 at 11:32 AM, Eugene Kirpichov <ki...@google.com>
>>> wrote:
>>>
>>>> Btw do you think windowing also should be separate from a PCollection?
>>>> I'm not sure whether it should be considered operational or not.
>>>>
>>>
>>> Interesting question. I think that windowing is more coupled to a
>>> PCollection, but I still mull this over once in a while. Here are some
>>> fairly unedited thoughts:
>>>
>>> Ignoring merging, one perspective is that the window is just a key with
>>> a deadline.
>>>
>>
>> That is only true when performing an aggregation. Records can be
>> associated with a window, and do not require keys at that point. The
>> "deadline" only applies when something like a GBK is assigned.
>>
>
> Yea, that situation -- windows assigned but no aggregation yet -- is
> analogous to data being a KV prior to the GBK. The main function that
> windows actually serve in the life of data processing is to allow
> aggregations over unbounded data with bounded resources. Only aggregation
> really needs them - if you just have a pass-through sequence of ParDos
> windows don't really do anything.
>
I disagree. There are multiple instances where windowing is used without an
aggregation after. Fundamentally windowing is a function on elements. This
function is used during aggregations to bound aggregations, but makes sense
on its own. Thinking of windowing as a "timeout" makes for an intuitive
model, but I don't think it's really the right model. For one thing, that
intuitive model makes less sense in batch.
> Kenn
>
>
>> From this perspective, the distinction between key and window is not
>>> important; you could just say that GBK requires the composite key for a
>>> group to eventually expire (in SQL terms, you just need one of the GROUP BY
>>> arguments to provide the deadline, and they are otherwise all on equal
>>> footing). And so the window is just as much a part of the data as the key.
>>> Without merging, once it is assigned you don't need to keep around the
>>> WindowFn or any such. Of course, our way of automatically propagating
>>> windows from inputs to outputs, akin to making MapValues the default mode
>>> of computation, requires the window to be a distinguished secondary key.
>>>
>>> Another way I think about it is that the windowing + watermark + allowed
>>> lateness defines which elements are a part of a PCollection and which are
>>> not. Dropped data semantically never existed in the first place. This was
>>> actually independent of windowing before the "window expiration" model of
>>> dropping data. I still think window expiration + GC + dropping go together
>>> nicely, and drop less data needlessly, but just dropping data behind the
>>> watermark + allowed lateness has some appeal for isolating the operational
>>> aspect here.
>>>
>>> Operationally, you might take the view that the act of expiration and
>>> dropping all remaining data is a configuration on the GBK. Then the
>>> WindowingStrategy, like windows and KV, are plumbing devices to reach a GBK
>>> that may be deep in a composite (which is certainly true anyhow). I don't
>>> really like this, because I would like the output of a GBK to be a
>>> straightforward function of its input - in the unbounded case I would like
>>> to be specified as having to agree with the bounded spec for any finite
>>> prefix. I'm not sure if an operational view is amenable to this. If they
>>> both work, then being able to switch perspectives back and forth would be
>>> cool.
>>>
>>> I think there are some inconsistencies in the above intuitions, and then
>>> there's merging...
>>>
>>> Kenn
>>>
>>>
>>> Also, I think anyone reading this document really ought to at least skim
>>>> the (linked from there) http://s.apache.org/beam-streams-tables and
>>>> internalize the idea of "PCollections as changelogs, aggregations as tables
>>>> on which the changelog acts". It probably would be good to rewrite our
>>>> documentation with this in mind: even with my experience on the Beam team,
>>>> this simple idea made it much easier for me to think clearly about all the
>>>> concepts.
>>>>
>>>> I'm very excited about both of these ideas, I think they rival in
>>>> importance the idea of batch/streaming unification and will end up being a
>>>> fundamental part of the future of Beam model.
>>>>
>>>> On Thu, Nov 30, 2017 at 8:52 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> wrote:
>>>>
>>>>> Hi Kenn,
>>>>>
>>>>> very interesting idea. It sounds more usable and "logic".
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 11/30/2017 09:06 PM, Kenneth Knowles wrote:
>>>>> > Hi all,
>>>>> >
>>>>> > Triggers are one of the more novel aspects of Beam's support for
>>>>> unbounded data.
>>>>> > They are also one of the most challenging aspects of the model.
>>>>> >
>>>>> > Ben & I have been working on a major new idea for how triggers could
>>>>> work in the
>>>>> > Beam model. We think it will make triggers much more usable, create
>>>>> new
>>>>> > opportunities for no-knobs execution/optimization, and improve
>>>>> compatibility
>>>>> > with DSLs like SQL. (also eliminate a whole class of bugs)
>>>>> >
>>>>> > Triggering is for sinks!
>>>>> >
>>>>> > https://s.apache.org/beam-sink-triggers
>>>>> >
>>>>> > Please take a look at this "1"-pager and give feedback.
>>>>> >
>>>>> > Kenn
>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbonofre@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Kenneth Knowles <kl...@google.com>.
On Tue, Dec 5, 2017 at 1:50 PM, Reuven Lax <re...@google.com> wrote:
>
>
> On Tue, Dec 5, 2017 at 12:00 PM, Kenneth Knowles <kl...@google.com> wrote:
>
>> On Tue, Dec 5, 2017 at 11:32 AM, Eugene Kirpichov <ki...@google.com>
>> wrote:
>>
>>> Btw do you think windowing also should be separate from a PCollection?
>>> I'm not sure whether it should be considered operational or not.
>>>
>>
>> Interesting question. I think that windowing is more coupled to a
>> PCollection, but I still mull this over once in a while. Here are some
>> fairly unedited thoughts:
>>
>> Ignoring merging, one perspective is that the window is just a key with a
>> deadline.
>>
>
> That is only true when performing an aggregation. Records can be
> associated with a window, and do not require keys at that point. The
> "deadline" only applies when something like a GBK is assigned.
>
Yea, that situation -- windows assigned but no aggregation yet -- is
analogous to data being a KV prior to the GBK. The main function that
windows actually serve in the life of data processing is to allow
aggregations over unbounded data with bounded resources. Only aggregation
really needs them - if you just have a pass-through sequence of ParDos
windows don't really do anything.
Kenn
> From this perspective, the distinction between key and window is not
>> important; you could just say that GBK requires the composite key for a
>> group to eventually expire (in SQL terms, you just need one of the GROUP BY
>> arguments to provide the deadline, and they are otherwise all on equal
>> footing). And so the window is just as much a part of the data as the key.
>> Without merging, once it is assigned you don't need to keep around the
>> WindowFn or any such. Of course, our way of automatically propagating
>> windows from inputs to outputs, akin to making MapValues the default mode
>> of computation, requires the window to be a distinguished secondary key.
>>
>> Another way I think about it is that the windowing + watermark + allowed
>> lateness defines which elements are a part of a PCollection and which are
>> not. Dropped data semantically never existed in the first place. This was
>> actually independent of windowing before the "window expiration" model of
>> dropping data. I still think window expiration + GC + dropping go together
>> nicely, and drop less data needlessly, but just dropping data behind the
>> watermark + allowed lateness has some appeal for isolating the operational
>> aspect here.
>>
>> Operationally, you might take the view that the act of expiration and
>> dropping all remaining data is a configuration on the GBK. Then the
>> WindowingStrategy, like windows and KV, are plumbing devices to reach a GBK
>> that may be deep in a composite (which is certainly true anyhow). I don't
>> really like this, because I would like the output of a GBK to be a
>> straightforward function of its input - in the unbounded case I would like
>> to be specified as having to agree with the bounded spec for any finite
>> prefix. I'm not sure if an operational view is amenable to this. If they
>> both work, then being able to switch perspectives back and forth would be
>> cool.
>>
>> I think there are some inconsistencies in the above intuitions, and then
>> there's merging...
>>
>> Kenn
>>
>>
>> Also, I think anyone reading this document really ought to at least skim
>>> the (linked from there) http://s.apache.org/beam-streams-tables and
>>> internalize the idea of "PCollections as changelogs, aggregations as tables
>>> on which the changelog acts". It probably would be good to rewrite our
>>> documentation with this in mind: even with my experience on the Beam team,
>>> this simple idea made it much easier for me to think clearly about all the
>>> concepts.
>>>
>>> I'm very excited about both of these ideas, I think they rival in
>>> importance the idea of batch/streaming unification and will end up being a
>>> fundamental part of the future of Beam model.
>>>
>>> On Thu, Nov 30, 2017 at 8:52 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Hi Kenn,
>>>>
>>>> very interesting idea. It sounds more usable and "logic".
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 11/30/2017 09:06 PM, Kenneth Knowles wrote:
>>>> > Hi all,
>>>> >
>>>> > Triggers are one of the more novel aspects of Beam's support for
>>>> unbounded data.
>>>> > They are also one of the most challenging aspects of the model.
>>>> >
>>>> > Ben & I have been working on a major new idea for how triggers could
>>>> work in the
>>>> > Beam model. We think it will make triggers much more usable, create
>>>> new
>>>> > opportunities for no-knobs execution/optimization, and improve
>>>> compatibility
>>>> > with DSLs like SQL. (also eliminate a whole class of bugs)
>>>> >
>>>> > Triggering is for sinks!
>>>> >
>>>> > https://s.apache.org/beam-sink-triggers
>>>> >
>>>> > Please take a look at this "1"-pager and give feedback.
>>>> >
>>>> > Kenn
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>>>
>>>
>>
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Reuven Lax <re...@google.com>.
On Tue, Dec 5, 2017 at 12:00 PM, Kenneth Knowles <kl...@google.com> wrote:
> On Tue, Dec 5, 2017 at 11:32 AM, Eugene Kirpichov <ki...@google.com>
> wrote:
>
>> Btw do you think windowing also should be separate from a PCollection?
>> I'm not sure whether it should be considered operational or not.
>>
>
> Interesting question. I think that windowing is more coupled to a
> PCollection, but I still mull this over once in a while. Here are some
> fairly unedited thoughts:
>
> Ignoring merging, one perspective is that the window is just a key with a
> deadline.
>
That is only true when performing an aggregation. Records can be associated
with a window, and do not require keys at that point. The "deadline" only
applies when something like a GBK is assigned.
From this perspective, the distinction between key and window is not
> important; you could just say that GBK requires the composite key for a
> group to eventually expire (in SQL terms, you just need one of the GROUP BY
> arguments to provide the deadline, and they are otherwise all on equal
> footing). And so the window is just as much a part of the data as the key.
> Without merging, once it is assigned you don't need to keep around the
> WindowFn or any such. Of course, our way of automatically propagating
> windows from inputs to outputs, akin to making MapValues the default mode
> of computation, requires the window to be a distinguished secondary key.
>
> Another way I think about it is that the windowing + watermark + allowed
> lateness defines which elements are a part of a PCollection and which are
> not. Dropped data semantically never existed in the first place. This was
> actually independent of windowing before the "window expiration" model of
> dropping data. I still think window expiration + GC + dropping go together
> nicely, and drop less data needlessly, but just dropping data behind the
> watermark + allowed lateness has some appeal for isolating the operational
> aspect here.
>
> Operationally, you might take the view that the act of expiration and
> dropping all remaining data is a configuration on the GBK. Then the
> WindowingStrategy, like windows and KV, are plumbing devices to reach a GBK
> that may be deep in a composite (which is certainly true anyhow). I don't
> really like this, because I would like the output of a GBK to be a
> straightforward function of its input - in the unbounded case I would like
> to be specified as having to agree with the bounded spec for any finite
> prefix. I'm not sure if an operational view is amenable to this. If they
> both work, then being able to switch perspectives back and forth would be
> cool.
>
> I think there are some inconsistencies in the above intuitions, and then
> there's merging...
>
> Kenn
>
>
> Also, I think anyone reading this document really ought to at least skim
>> the (linked from there) http://s.apache.org/beam-streams-tables and
>> internalize the idea of "PCollections as changelogs, aggregations as tables
>> on which the changelog acts". It probably would be good to rewrite our
>> documentation with this in mind: even with my experience on the Beam team,
>> this simple idea made it much easier for me to think clearly about all the
>> concepts.
>>
>> I'm very excited about both of these ideas, I think they rival in
>> importance the idea of batch/streaming unification and will end up being a
>> fundamental part of the future of Beam model.
>>
>> On Thu, Nov 30, 2017 at 8:52 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Hi Kenn,
>>>
>>> very interesting idea. It sounds more usable and "logic".
>>>
>>> Regards
>>> JB
>>>
>>> On 11/30/2017 09:06 PM, Kenneth Knowles wrote:
>>> > Hi all,
>>> >
>>> > Triggers are one of the more novel aspects of Beam's support for
>>> unbounded data.
>>> > They are also one of the most challenging aspects of the model.
>>> >
>>> > Ben & I have been working on a major new idea for how triggers could
>>> work in the
>>> > Beam model. We think it will make triggers much more usable, create new
>>> > opportunities for no-knobs execution/optimization, and improve
>>> compatibility
>>> > with DSLs like SQL. (also eliminate a whole class of bugs)
>>> >
>>> > Triggering is for sinks!
>>> >
>>> > https://s.apache.org/beam-sink-triggers
>>> >
>>> > Please take a look at this "1"-pager and give feedback.
>>> >
>>> > Kenn
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Kenneth Knowles <kl...@google.com>.
On Tue, Dec 5, 2017 at 11:32 AM, Eugene Kirpichov <ki...@google.com>
wrote:
> Btw do you think windowing also should be separate from a PCollection? I'm
> not sure whether it should be considered operational or not.
>
Interesting question. I think that windowing is more coupled to a
PCollection, but I still mull this over once in a while. Here are some
fairly unedited thoughts:
Ignoring merging, one perspective is that the window is just a key with a
deadline. From this perspective, the distinction between key and window is
not important; you could just say that GBK requires the composite key for a
group to eventually expire (in SQL terms, you just need one of the GROUP BY
arguments to provide the deadline, and they are otherwise all on equal
footing). And so the window is just as much a part of the data as the key.
Without merging, once it is assigned you don't need to keep around the
WindowFn or any such. Of course, our way of automatically propagating
windows from inputs to outputs, akin to making MapValues the default mode
of computation, requires the window to be a distinguished secondary key.
Another way I think about it is that the windowing + watermark + allowed
lateness defines which elements are a part of a PCollection and which are
not. Dropped data semantically never existed in the first place. This was
actually independent of windowing before the "window expiration" model of
dropping data. I still think window expiration + GC + dropping go together
nicely, and drop less data needlessly, but just dropping data behind the
watermark + allowed lateness has some appeal for isolating the operational
aspect here.
Operationally, you might take the view that the act of expiration and
dropping all remaining data is a configuration on the GBK. Then the
WindowingStrategy, like windows and KV, are plumbing devices to reach a GBK
that may be deep in a composite (which is certainly true anyhow). I don't
really like this, because I would like the output of a GBK to be a
straightforward function of its input - in the unbounded case I would like
to be specified as having to agree with the bounded spec for any finite
prefix. I'm not sure if an operational view is amenable to this. If they
both work, then being able to switch perspectives back and forth would be
cool.
I think there are some inconsistencies in the above intuitions, and then
there's merging...
Kenn
Also, I think anyone reading this document really ought to at least skim
> the (linked from there) http://s.apache.org/beam-streams-tables and
> internalize the idea of "PCollections as changelogs, aggregations as tables
> on which the changelog acts". It probably would be good to rewrite our
> documentation with this in mind: even with my experience on the Beam team,
> this simple idea made it much easier for me to think clearly about all the
> concepts.
>
> I'm very excited about both of these ideas, I think they rival in
> importance the idea of batch/streaming unification and will end up being a
> fundamental part of the future of Beam model.
>
> On Thu, Nov 30, 2017 at 8:52 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi Kenn,
>>
>> very interesting idea. It sounds more usable and "logic".
>>
>> Regards
>> JB
>>
>> On 11/30/2017 09:06 PM, Kenneth Knowles wrote:
>> > Hi all,
>> >
>> > Triggers are one of the more novel aspects of Beam's support for
>> unbounded data.
>> > They are also one of the most challenging aspects of the model.
>> >
>> > Ben & I have been working on a major new idea for how triggers could
>> work in the
>> > Beam model. We think it will make triggers much more usable, create new
>> > opportunities for no-knobs execution/optimization, and improve
>> compatibility
>> > with DSLs like SQL. (also eliminate a whole class of bugs)
>> >
>> > Triggering is for sinks!
>> >
>> > https://s.apache.org/beam-sink-triggers
>> >
>> > Please take a look at this "1"-pager and give feedback.
>> >
>> > Kenn
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Eugene Kirpichov <ki...@google.com>.
Huge +1. I think this is a wonderful idea that, if implemented, will make
major improvements in many areas of Beam:
- As you said, eliminating a class of data loss bugs and highly unintuitive
behaviors or errors (eg CoGbk)
- As per the thread "Callbacks/other functions run after a PDone/output
transform", your document also gives a conceptual framework under which to
think about the interaction of sinks and triggers, which enables sensible
semantics for "do X after Y is done"
- Most importantly, it just makes sense :) unlike current triggers, which
unintuitively say that triggering is something inherent to a PCollection
rather than to a processing step (triggers are clearly an operational
device). Btw do you think windowing also should be separate from a
PCollection? I'm not sure whether it should be considered operational or
not.
Also, I think anyone reading this document really ought to at least skim
the (linked from there) http://s.apache.org/beam-streams-tables and
internalize the idea of "PCollections as changelogs, aggregations as tables
on which the changelog acts". It probably would be good to rewrite our
documentation with this in mind: even with my experience on the Beam team,
this simple idea made it much easier for me to think clearly about all the
concepts.
I'm very excited about both of these ideas, I think they rival in
importance the idea of batch/streaming unification and will end up being a
fundamental part of the future of Beam model.
On Thu, Nov 30, 2017 at 8:52 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Kenn,
>
> very interesting idea. It sounds more usable and "logic".
>
> Regards
> JB
>
> On 11/30/2017 09:06 PM, Kenneth Knowles wrote:
> > Hi all,
> >
> > Triggers are one of the more novel aspects of Beam's support for
> unbounded data.
> > They are also one of the most challenging aspects of the model.
> >
> > Ben & I have been working on a major new idea for how triggers could
> work in the
> > Beam model. We think it will make triggers much more usable, create new
> > opportunities for no-knobs execution/optimization, and improve
> compatibility
> > with DSLs like SQL. (also eliminate a whole class of bugs)
> >
> > Triggering is for sinks!
> >
> > https://s.apache.org/beam-sink-triggers
> >
> > Please take a look at this "1"-pager and give feedback.
> >
> > Kenn
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [DISCUSS] [DOC] Triggering is for sinks!
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Kenn,
very interesting idea. It sounds more usable and "logic".
Regards
JB
On 11/30/2017 09:06 PM, Kenneth Knowles wrote:
> Hi all,
>
> Triggers are one of the more novel aspects of Beam's support for unbounded data.
> They are also one of the most challenging aspects of the model.
>
> Ben & I have been working on a major new idea for how triggers could work in the
> Beam model. We think it will make triggers much more usable, create new
> opportunities for no-knobs execution/optimization, and improve compatibility
> with DSLs like SQL. (also eliminate a whole class of bugs)
>
> Triggering is for sinks!
>
> https://s.apache.org/beam-sink-triggers
>
> Please take a look at this "1"-pager and give feedback.
>
> Kenn
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com