You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Manu Zhang <ow...@gmail.com> on 2016/12/01 04:08:12 UTC
Event-time based in-window trigger
Hi,
Recently I’m addressing a problem where users want to trigger after
watermark past each element (i.e. in the middle of event-time window). I
fail to find an existing trigger that does so. Any idea on model this
problem with Beam ?
Thanks,
Manu Zhang
Re: Event-time based in-window trigger
Posted by Manu Zhang <ow...@gmail.com>.
created BEAM-1119 <https://issues.apache.org/jira/browse/BEAM-1119> and
assigned to you.
On Fri, Dec 9, 2016 at 11:51 AM Kenneth Knowles <kl...@google.com> wrote:
> I don't think there is a JIRA for this, but I agree that the use case is
> good and compatible with the model. Please do open a JIRA ticket in the
> beam-model component. Feel free to assign to me or leave unassigned.
>
> On Thu, Dec 8, 2016 at 5:30 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> @Kenn and @Tyler,
>
> Given the use case is defined (hope I've explained it clearly), do we have
> plans/jiras to add the new functionality ?
>
> Thanks,
> Manu
>
> On Fri, Dec 2, 2016 at 12:21 PM Manu Zhang <ow...@gmail.com>
> wrote:
>
> @Kenn,
>
> 1. when the watermark jumps from 0 to 7, http://foo -> http://foo/bar ->
> http://foo will be emitted
> We can emit events with timestamps before watermark in the pane
> 2. http://foo -> http://foo/bizzle -> http://foo/bar -> http://foo will
> be emitted if it's within the allowed lateness
> which Beam already allows us to do.
>
> To elaborate on the use case, when users are visiting Amazon we want to
> offer them best recommendations.
> Thus, we would like to know what leads to their final decision and track
> the pages they visit until clicking the "Add to cart" button.
> It will be too late if we only send the results when they finish shopping.
>
> @Tyler,
> I don't think it's likely to happen for my use case. Think about a user
> jumping between pages like crazy. Meanwhile, we can control how fast
> watermark progresses as long as it meets the latency requirement.
>
>
>
>
> On Fri, Dec 2, 2016 at 11:45 AM Tyler Akidau <ta...@google.com> wrote:
>
> And one more question while we're at it: what if you have events happening
> every second within the window? Do you really want to emit a new pane every
> second as the watermark progresses (assuming it progresses relatively
> smoothly)? What if we're talking differences of event times of
> milliseconds? Is one pane per millisecond what you want?
>
> -Tyler
>
> On Fri, Dec 2, 2016 at 10:41 AM Kenneth Knowles <kl...@google.com> wrote:
>
> Thanks for laying out some details.
>
> On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Yes, the difficulty is to define that trigger. The existing triggers fire
> at the end of window. (I could be mistaken, which will be good news)
>
>
> You are not mistaken that the only existing event time trigger is the one
> that fires at the end of the window. The trigger you describe would be a
> new primitive trigger. It fits with the design, if we ensure monotonicity,
> etc. Actually implementing it in the backend is easy, of course. We
> actually had something like it, but didn't quite nail it down so we removed
> it until we had a solid use case and design for it.
>
> B and C which are not mutually exclusive
> More on my use case. Say a user visits http://foo at 1, http://foo/bar at
> 4 and back to http://foo at 5 all in a Session
> we would want to emit
>
> http://foo when the watermark passes 1
> http://foo -> http://foo/bar when the watermark passes 4
> http://foo -> http://foo/bar -> http://foo when the watermark passes 5
>
>
> What would you want to emit when the watermark jumps from 0 to 7 and all
> three of the above are buffered?
>
> What would you want to emit when the watermark was at 9 and
> http://foo/bizzle came in with timestamp 3?
>
> Kenn
>
>
>
>
> On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <bc...@google.com> wrote:
>
> As a clarifying question:
>
> If you have three elements in the pane with timestamps [1, 4, 5], would
> you:
> A. want to emit that entire pane when the watermark passes 1
> B. want to emit that entire pane when the watermark passes 5
> C. emit a fragment of that pane containing only the first element when the
> watermark passes 1
>
> On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <ta...@google.com> wrote:
>
> So what you want is essentially a trigger that fires when the watermark
> has passed the event time of the oldest un-emitted element in the current
> pane? You could them presumably wrap this in a repeat to get the overall
> desired semantics, right?
>
> -Tyler
>
>
> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <ow...@gmail.com> wrote:
>
> My use case is to track user trajectory based on page view event when they
> visit a website. The input would be like a list of PageView(userId, url,
> eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying
> Sessions with event time trigger. Note we can't wait for the end of session
> window due to latency requirement. Instead, we want to emit the user
> trajectories whenever a buffered PageView's event time is passed by
> watermark.
>
> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
>
> Can you provide more details about the problem your trying to solve with
> some examples showing input and the expected output?
>
>
>
>
> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Hi,
>
> Recently I’m addressing a problem where users want to trigger after
> watermark past each element (i.e. in the middle of event-time window). I
> fail to find an existing trigger that does so. Any idea on model this
> problem with Beam ?
>
> Thanks,
> Manu Zhang
>
>
>
>
Re: Event-time based in-window trigger
Posted by Kenneth Knowles <kl...@google.com>.
I don't think there is a JIRA for this, but I agree that the use case is
good and compatible with the model. Please do open a JIRA ticket in the
beam-model component. Feel free to assign to me or leave unassigned.
On Thu, Dec 8, 2016 at 5:30 PM, Manu Zhang <ow...@gmail.com> wrote:
> @Kenn and @Tyler,
>
> Given the use case is defined (hope I've explained it clearly), do we have
> plans/jiras to add the new functionality ?
>
> Thanks,
> Manu
>
> On Fri, Dec 2, 2016 at 12:21 PM Manu Zhang <ow...@gmail.com>
> wrote:
>
>> @Kenn,
>>
>> 1. when the watermark jumps from 0 to 7, http://foo -> http://foo/bar ->
>> http://foo will be emitted
>> We can emit events with timestamps before watermark in the pane
>> 2. http://foo -> http://foo/bizzle -> http://foo/bar -> http://foo will
>> be emitted if it's within the allowed lateness
>> which Beam already allows us to do.
>>
>> To elaborate on the use case, when users are visiting Amazon we want to
>> offer them best recommendations.
>> Thus, we would like to know what leads to their final decision and track
>> the pages they visit until clicking the "Add to cart" button.
>> It will be too late if we only send the results when they finish shopping.
>>
>> @Tyler,
>> I don't think it's likely to happen for my use case. Think about a user
>> jumping between pages like crazy. Meanwhile, we can control how fast
>> watermark progresses as long as it meets the latency requirement.
>>
>>
>>
>>
>> On Fri, Dec 2, 2016 at 11:45 AM Tyler Akidau <ta...@google.com> wrote:
>>
>> And one more question while we're at it: what if you have events
>> happening every second within the window? Do you really want to emit a new
>> pane every second as the watermark progresses (assuming it progresses
>> relatively smoothly)? What if we're talking differences of event times of
>> milliseconds? Is one pane per millisecond what you want?
>>
>> -Tyler
>>
>> On Fri, Dec 2, 2016 at 10:41 AM Kenneth Knowles <kl...@google.com> wrote:
>>
>> Thanks for laying out some details.
>>
>> On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <ow...@gmail.com>
>> wrote:
>>
>> Yes, the difficulty is to define that trigger. The existing triggers fire
>> at the end of window. (I could be mistaken, which will be good news)
>>
>>
>> You are not mistaken that the only existing event time trigger is the one
>> that fires at the end of the window. The trigger you describe would be a
>> new primitive trigger. It fits with the design, if we ensure monotonicity,
>> etc. Actually implementing it in the backend is easy, of course. We
>> actually had something like it, but didn't quite nail it down so we removed
>> it until we had a solid use case and design for it.
>>
>> B and C which are not mutually exclusive
>> More on my use case. Say a user visits http://foo at 1, http://foo/bar at
>> 4 and back to http://foo at 5 all in a Session
>> we would want to emit
>>
>> http://foo when the watermark passes 1
>> http://foo -> http://foo/bar when the watermark passes 4
>> http://foo -> http://foo/bar -> http://foo when the watermark passes 5
>>
>>
>> What would you want to emit when the watermark jumps from 0 to 7 and all
>> three of the above are buffered?
>>
>> What would you want to emit when the watermark was at 9 and
>> http://foo/bizzle came in with timestamp 3?
>>
>> Kenn
>>
>>
>>
>>
>> On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <bc...@google.com>
>> wrote:
>>
>> As a clarifying question:
>>
>> If you have three elements in the pane with timestamps [1, 4, 5], would
>> you:
>> A. want to emit that entire pane when the watermark passes 1
>> B. want to emit that entire pane when the watermark passes 5
>> C. emit a fragment of that pane containing only the first element when
>> the watermark passes 1
>>
>> On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <ta...@google.com> wrote:
>>
>> So what you want is essentially a trigger that fires when the watermark
>> has passed the event time of the oldest un-emitted element in the current
>> pane? You could them presumably wrap this in a repeat to get the overall
>> desired semantics, right?
>>
>> -Tyler
>>
>>
>> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <ow...@gmail.com>
>> wrote:
>>
>> My use case is to track user trajectory based on page view event when
>> they visit a website. The input would be like a list of
>> PageView(userId, url, eventTimestamp) with watermarks (= eventTimestamp -
>> duration). I'm trying Sessions with event time trigger. Note we can't wait
>> for the end of session window due to latency requirement. Instead, we want
>> to emit the user trajectories whenever a buffered PageView's event time is
>> passed by watermark.
>>
>> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>> Can you provide more details about the problem your trying to solve with
>> some examples showing input and the expected output?
>>
>>
>>
>>
>> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> Recently I’m addressing a problem where users want to trigger after
>> watermark past each element (i.e. in the middle of event-time window). I
>> fail to find an existing trigger that does so. Any idea on model this
>> problem with Beam ?
>>
>> Thanks,
>> Manu Zhang
>>
>>
>>
Re: Event-time based in-window trigger
Posted by Manu Zhang <ow...@gmail.com>.
@Kenn and @Tyler,
Given the use case is defined (hope I've explained it clearly), do we have
plans/jiras to add the new functionality ?
Thanks,
Manu
On Fri, Dec 2, 2016 at 12:21 PM Manu Zhang <ow...@gmail.com> wrote:
> @Kenn,
>
> 1. when the watermark jumps from 0 to 7, http://foo -> http://foo/bar ->
> http://foo will be emitted
> We can emit events with timestamps before watermark in the pane
> 2. http://foo -> http://foo/bizzle -> http://foo/bar -> http://foo will
> be emitted if it's within the allowed lateness
> which Beam already allows us to do.
>
> To elaborate on the use case, when users are visiting Amazon we want to
> offer them best recommendations.
> Thus, we would like to know what leads to their final decision and track
> the pages they visit until clicking the "Add to cart" button.
> It will be too late if we only send the results when they finish shopping.
>
> @Tyler,
> I don't think it's likely to happen for my use case. Think about a user
> jumping between pages like crazy. Meanwhile, we can control how fast
> watermark progresses as long as it meets the latency requirement.
>
>
>
>
> On Fri, Dec 2, 2016 at 11:45 AM Tyler Akidau <ta...@google.com> wrote:
>
> And one more question while we're at it: what if you have events happening
> every second within the window? Do you really want to emit a new pane every
> second as the watermark progresses (assuming it progresses relatively
> smoothly)? What if we're talking differences of event times of
> milliseconds? Is one pane per millisecond what you want?
>
> -Tyler
>
> On Fri, Dec 2, 2016 at 10:41 AM Kenneth Knowles <kl...@google.com> wrote:
>
> Thanks for laying out some details.
>
> On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Yes, the difficulty is to define that trigger. The existing triggers fire
> at the end of window. (I could be mistaken, which will be good news)
>
>
> You are not mistaken that the only existing event time trigger is the one
> that fires at the end of the window. The trigger you describe would be a
> new primitive trigger. It fits with the design, if we ensure monotonicity,
> etc. Actually implementing it in the backend is easy, of course. We
> actually had something like it, but didn't quite nail it down so we removed
> it until we had a solid use case and design for it.
>
> B and C which are not mutually exclusive
> More on my use case. Say a user visits http://foo at 1, http://foo/bar at
> 4 and back to http://foo at 5 all in a Session
> we would want to emit
>
> http://foo when the watermark passes 1
> http://foo -> http://foo/bar when the watermark passes 4
> http://foo -> http://foo/bar -> http://foo when the watermark passes 5
>
>
> What would you want to emit when the watermark jumps from 0 to 7 and all
> three of the above are buffered?
>
> What would you want to emit when the watermark was at 9 and
> http://foo/bizzle came in with timestamp 3?
>
> Kenn
>
>
>
>
> On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <bc...@google.com> wrote:
>
> As a clarifying question:
>
> If you have three elements in the pane with timestamps [1, 4, 5], would
> you:
> A. want to emit that entire pane when the watermark passes 1
> B. want to emit that entire pane when the watermark passes 5
> C. emit a fragment of that pane containing only the first element when the
> watermark passes 1
>
> On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <ta...@google.com> wrote:
>
> So what you want is essentially a trigger that fires when the watermark
> has passed the event time of the oldest un-emitted element in the current
> pane? You could them presumably wrap this in a repeat to get the overall
> desired semantics, right?
>
> -Tyler
>
>
> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <ow...@gmail.com> wrote:
>
> My use case is to track user trajectory based on page view event when they
> visit a website. The input would be like a list of PageView(userId, url,
> eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying
> Sessions with event time trigger. Note we can't wait for the end of session
> window due to latency requirement. Instead, we want to emit the user
> trajectories whenever a buffered PageView's event time is passed by
> watermark.
>
> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
>
> Can you provide more details about the problem your trying to solve with
> some examples showing input and the expected output?
>
>
>
>
> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Hi,
>
> Recently I’m addressing a problem where users want to trigger after
> watermark past each element (i.e. in the middle of event-time window). I
> fail to find an existing trigger that does so. Any idea on model this
> problem with Beam ?
>
> Thanks,
> Manu Zhang
>
>
>
Re: Event-time based in-window trigger
Posted by Manu Zhang <ow...@gmail.com>.
@Kenn,
1. when the watermark jumps from 0 to 7, http://foo -> http://foo/bar ->
http://foo will be emitted
We can emit events with timestamps before watermark in the pane
2. http://foo -> http://foo/bizzle -> http://foo/bar -> http://foo will be
emitted if it's within the allowed lateness
which Beam already allows us to do.
To elaborate on the use case, when users are visiting Amazon we want to
offer them best recommendations.
Thus, we would like to know what leads to their final decision and track
the pages they visit until clicking the "Add to cart" button.
It will be too late if we only send the results when they finish shopping.
@Tyler,
I don't think it's likely to happen for my use case. Think about a user
jumping between pages like crazy. Meanwhile, we can control how fast
watermark progresses as long as it meets the latency requirement.
On Fri, Dec 2, 2016 at 11:45 AM Tyler Akidau <ta...@google.com> wrote:
> And one more question while we're at it: what if you have events happening
> every second within the window? Do you really want to emit a new pane every
> second as the watermark progresses (assuming it progresses relatively
> smoothly)? What if we're talking differences of event times of
> milliseconds? Is one pane per millisecond what you want?
>
> -Tyler
>
> On Fri, Dec 2, 2016 at 10:41 AM Kenneth Knowles <kl...@google.com> wrote:
>
> Thanks for laying out some details.
>
> On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Yes, the difficulty is to define that trigger. The existing triggers fire
> at the end of window. (I could be mistaken, which will be good news)
>
>
> You are not mistaken that the only existing event time trigger is the one
> that fires at the end of the window. The trigger you describe would be a
> new primitive trigger. It fits with the design, if we ensure monotonicity,
> etc. Actually implementing it in the backend is easy, of course. We
> actually had something like it, but didn't quite nail it down so we removed
> it until we had a solid use case and design for it.
>
> B and C which are not mutually exclusive
> More on my use case. Say a user visits http://foo at 1, http://foo/bar at
> 4 and back to http://foo at 5 all in a Session
> we would want to emit
>
> http://foo when the watermark passes 1
> http://foo -> http://foo/bar when the watermark passes 4
> http://foo -> http://foo/bar -> http://foo when the watermark passes 5
>
>
> What would you want to emit when the watermark jumps from 0 to 7 and all
> three of the above are buffered?
>
> What would you want to emit when the watermark was at 9 and
> http://foo/bizzle came in with timestamp 3?
>
> Kenn
>
>
>
>
> On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <bc...@google.com> wrote:
>
> As a clarifying question:
>
> If you have three elements in the pane with timestamps [1, 4, 5], would
> you:
> A. want to emit that entire pane when the watermark passes 1
> B. want to emit that entire pane when the watermark passes 5
> C. emit a fragment of that pane containing only the first element when the
> watermark passes 1
>
> On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <ta...@google.com> wrote:
>
> So what you want is essentially a trigger that fires when the watermark
> has passed the event time of the oldest un-emitted element in the current
> pane? You could them presumably wrap this in a repeat to get the overall
> desired semantics, right?
>
> -Tyler
>
>
> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <ow...@gmail.com> wrote:
>
> My use case is to track user trajectory based on page view event when they
> visit a website. The input would be like a list of PageView(userId, url,
> eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying
> Sessions with event time trigger. Note we can't wait for the end of session
> window due to latency requirement. Instead, we want to emit the user
> trajectories whenever a buffered PageView's event time is passed by
> watermark.
>
> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
>
> Can you provide more details about the problem your trying to solve with
> some examples showing input and the expected output?
>
>
>
>
> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Hi,
>
> Recently I’m addressing a problem where users want to trigger after
> watermark past each element (i.e. in the middle of event-time window). I
> fail to find an existing trigger that does so. Any idea on model this
> problem with Beam ?
>
> Thanks,
> Manu Zhang
>
>
>
Re: Event-time based in-window trigger
Posted by Tyler Akidau <ta...@google.com>.
And one more question while we're at it: what if you have events happening
every second within the window? Do you really want to emit a new pane every
second as the watermark progresses (assuming it progresses relatively
smoothly)? What if we're talking differences of event times of
milliseconds? Is one pane per millisecond what you want?
-Tyler
On Fri, Dec 2, 2016 at 10:41 AM Kenneth Knowles <kl...@google.com> wrote:
> Thanks for laying out some details.
>
> On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Yes, the difficulty is to define that trigger. The existing triggers fire
> at the end of window. (I could be mistaken, which will be good news)
>
>
> You are not mistaken that the only existing event time trigger is the one
> that fires at the end of the window. The trigger you describe would be a
> new primitive trigger. It fits with the design, if we ensure monotonicity,
> etc. Actually implementing it in the backend is easy, of course. We
> actually had something like it, but didn't quite nail it down so we removed
> it until we had a solid use case and design for it.
>
> B and C which are not mutually exclusive
> More on my use case. Say a user visits http://foo at 1, http://foo/bar at
> 4 and back to http://foo at 5 all in a Session
> we would want to emit
>
> http://foo when the watermark passes 1
> http://foo -> http://foo/bar when the watermark passes 4
> http://foo -> http://foo/bar -> http://foo when the watermark passes 5
>
>
> What would you want to emit when the watermark jumps from 0 to 7 and all
> three of the above are buffered?
>
> What would you want to emit when the watermark was at 9 and
> http://foo/bizzle came in with timestamp 3?
>
> Kenn
>
>
>
>
> On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <bc...@google.com> wrote:
>
> As a clarifying question:
>
> If you have three elements in the pane with timestamps [1, 4, 5], would
> you:
> A. want to emit that entire pane when the watermark passes 1
> B. want to emit that entire pane when the watermark passes 5
> C. emit a fragment of that pane containing only the first element when the
> watermark passes 1
>
> On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <ta...@google.com> wrote:
>
> So what you want is essentially a trigger that fires when the watermark
> has passed the event time of the oldest un-emitted element in the current
> pane? You could them presumably wrap this in a repeat to get the overall
> desired semantics, right?
>
> -Tyler
>
>
> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <ow...@gmail.com> wrote:
>
> My use case is to track user trajectory based on page view event when they
> visit a website. The input would be like a list of PageView(userId, url,
> eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying
> Sessions with event time trigger. Note we can't wait for the end of session
> window due to latency requirement. Instead, we want to emit the user
> trajectories whenever a buffered PageView's event time is passed by
> watermark.
>
> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
>
> Can you provide more details about the problem your trying to solve with
> some examples showing input and the expected output?
>
>
>
>
> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Hi,
>
> Recently I’m addressing a problem where users want to trigger after
> watermark past each element (i.e. in the middle of event-time window). I
> fail to find an existing trigger that does so. Any idea on model this
> problem with Beam ?
>
> Thanks,
> Manu Zhang
>
>
>
Re: Event-time based in-window trigger
Posted by Kenneth Knowles <kl...@google.com>.
Thanks for laying out some details.
On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <ow...@gmail.com> wrote:
>
> Yes, the difficulty is to define that trigger. The existing triggers fire
> at the end of window. (I could be mistaken, which will be good news)
>
You are not mistaken that the only existing event time trigger is the one
that fires at the end of the window. The trigger you describe would be a
new primitive trigger. It fits with the design, if we ensure monotonicity,
etc. Actually implementing it in the backend is easy, of course. We
actually had something like it, but didn't quite nail it down so we removed
it until we had a solid use case and design for it.
B and C which are not mutually exclusive
> More on my use case. Say a user visits http://foo at 1, http://foo/bar at
> 4 and back to http://foo at 5 all in a Session
> we would want to emit
>
> http://foo when the watermark passes 1
> http://foo -> http://foo/bar when the watermark passes 4
> http://foo -> http://foo/bar -> http://foo when the watermark passes 5
>
What would you want to emit when the watermark jumps from 0 to 7 and all
three of the above are buffered?
What would you want to emit when the watermark was at 9 and
http://foo/bizzle came in with timestamp 3?
Kenn
>
>
> On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <bc...@google.com> wrote:
>
>> As a clarifying question:
>>
>> If you have three elements in the pane with timestamps [1, 4, 5], would
>> you:
>> A. want to emit that entire pane when the watermark passes 1
>> B. want to emit that entire pane when the watermark passes 5
>> C. emit a fragment of that pane containing only the first element when
>> the watermark passes 1
>>
>> On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <ta...@google.com> wrote:
>>
>> So what you want is essentially a trigger that fires when the watermark
>> has passed the event time of the oldest un-emitted element in the current
>> pane? You could them presumably wrap this in a repeat to get the overall
>> desired semantics, right?
>>
>> -Tyler
>>
>>
>> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <ow...@gmail.com>
>> wrote:
>>
>> My use case is to track user trajectory based on page view event when
>> they visit a website. The input would be like a list of
>> PageView(userId, url, eventTimestamp) with watermarks (= eventTimestamp -
>> duration). I'm trying Sessions with event time trigger. Note we can't wait
>> for the end of session window due to latency requirement. Instead, we want
>> to emit the user trajectories whenever a buffered PageView's event time is
>> passed by watermark.
>>
>> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>> Can you provide more details about the problem your trying to solve with
>> some examples showing input and the expected output?
>>
>>
>>
>>
>> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> Recently I’m addressing a problem where users want to trigger after
>> watermark past each element (i.e. in the middle of event-time window). I
>> fail to find an existing trigger that does so. Any idea on model this
>> problem with Beam ?
>>
>> Thanks,
>> Manu Zhang
>>
>>
>>
Re: Event-time based in-window trigger
Posted by Manu Zhang <ow...@gmail.com>.
@Tyler,
Yes, the difficulty is to define that trigger. The existing triggers fire
at the end of window. (I could be mistaken, which will be good news)
@Ben,
B and C which are not mutually exclusive
More on my use case. Say a user visits http://foo at 1, http://foo/bar at 4
and back to http://foo at 5 all in a Session
we would want to emit
http://foo when the watermark passes 1
http://foo -> http://foo/bar when the watermark passes 4
http://foo -> http://foo/bar -> http://foo when the watermark passes 5
On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <bc...@google.com> wrote:
> As a clarifying question:
>
> If you have three elements in the pane with timestamps [1, 4, 5], would
> you:
> A. want to emit that entire pane when the watermark passes 1
> B. want to emit that entire pane when the watermark passes 5
> C. emit a fragment of that pane containing only the first element when the
> watermark passes 1
>
> On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <ta...@google.com> wrote:
>
> So what you want is essentially a trigger that fires when the watermark
> has passed the event time of the oldest un-emitted element in the current
> pane? You could them presumably wrap this in a repeat to get the overall
> desired semantics, right?
>
> -Tyler
>
>
> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <ow...@gmail.com> wrote:
>
> My use case is to track user trajectory based on page view event when they
> visit a website. The input would be like a list of PageView(userId, url,
> eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying
> Sessions with event time trigger. Note we can't wait for the end of session
> window due to latency requirement. Instead, we want to emit the user
> trajectories whenever a buffered PageView's event time is passed by
> watermark.
>
> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
>
> Can you provide more details about the problem your trying to solve with
> some examples showing input and the expected output?
>
>
>
>
> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Hi,
>
> Recently I’m addressing a problem where users want to trigger after
> watermark past each element (i.e. in the middle of event-time window). I
> fail to find an existing trigger that does so. Any idea on model this
> problem with Beam ?
>
> Thanks,
> Manu Zhang
>
>
>
Re: Event-time based in-window trigger
Posted by Ben Chambers <bc...@google.com>.
As a clarifying question:
If you have three elements in the pane with timestamps [1, 4, 5], would you:
A. want to emit that entire pane when the watermark passes 1
B. want to emit that entire pane when the watermark passes 5
C. emit a fragment of that pane containing only the first element when the
watermark passes 1
On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <ta...@google.com> wrote:
> So what you want is essentially a trigger that fires when the watermark
> has passed the event time of the oldest un-emitted element in the current
> pane? You could them presumably wrap this in a repeat to get the overall
> desired semantics, right?
>
> -Tyler
>
>
> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <ow...@gmail.com> wrote:
>
> My use case is to track user trajectory based on page view event when they
> visit a website. The input would be like a list of PageView(userId, url,
> eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying
> Sessions with event time trigger. Note we can't wait for the end of session
> window due to latency requirement. Instead, we want to emit the user
> trajectories whenever a buffered PageView's event time is passed by
> watermark.
>
> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
>
> Can you provide more details about the problem your trying to solve with
> some examples showing input and the expected output?
>
>
>
>
> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Hi,
>
> Recently I’m addressing a problem where users want to trigger after
> watermark past each element (i.e. in the middle of event-time window). I
> fail to find an existing trigger that does so. Any idea on model this
> problem with Beam ?
>
> Thanks,
> Manu Zhang
>
>
>
Re: Event-time based in-window trigger
Posted by Tyler Akidau <ta...@google.com>.
So what you want is essentially a trigger that fires when the watermark has
passed the event time of the oldest un-emitted element in the current pane?
You could them presumably wrap this in a repeat to get the overall desired
semantics, right?
-Tyler
On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <ow...@gmail.com> wrote:
> My use case is to track user trajectory based on page view event when they
> visit a website. The input would be like a list of PageView(userId, url,
> eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying
> Sessions with event time trigger. Note we can't wait for the end of session
> window due to latency requirement. Instead, we want to emit the user
> trajectories whenever a buffered PageView's event time is passed by
> watermark.
>
> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
>
> Can you provide more details about the problem your trying to solve with
> some examples showing input and the expected output?
>
>
>
>
> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Hi,
>
> Recently I’m addressing a problem where users want to trigger after
> watermark past each element (i.e. in the middle of event-time window). I
> fail to find an existing trigger that does so. Any idea on model this
> problem with Beam ?
>
> Thanks,
> Manu Zhang
>
>
>
Re: Event-time based in-window trigger
Posted by Manu Zhang <ow...@gmail.com>.
My use case is to track user trajectory based on page view event when they
visit a website. The input would be like a list of PageView(userId, url,
eventTimestamp) with watermarks (= eventTimestamp - duration). I'm trying
Sessions with event time trigger. Note we can't wait for the end of session
window due to latency requirement. Instead, we want to emit the user
trajectories whenever a buffered PageView's event time is passed by
watermark.
On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <lc...@google.com> wrote:
> Can you provide more details about the problem your trying to solve with
> some examples showing input and the expected output?
>
>
>
>
> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
> wrote:
>
> Hi,
>
> Recently I’m addressing a problem where users want to trigger after
> watermark past each element (i.e. in the middle of event-time window). I
> fail to find an existing trigger that does so. Any idea on model this
> problem with Beam ?
>
> Thanks,
> Manu Zhang
>
>
>
Re: Event-time based in-window trigger
Posted by Lukasz Cwik <lc...@google.com>.
Can you provide more details about the problem your trying to solve with
some examples showing input and the expected output?
On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <ow...@gmail.com>
wrote:
> Hi,
>
> Recently I’m addressing a problem where users want to trigger after
> watermark past each element (i.e. in the middle of event-time window). I
> fail to find an existing trigger that does so. Any idea on model this
> problem with Beam ?
>
> Thanks,
> Manu Zhang
>