Sub-second Beam

[Aaron Dixon <at...@gmail.com>]

I've been building pipelines and benchmarking Beam jobs in Dataflow.

Without windowing, latencies look pretty good (reliably sub-second) from
ingest to sink.

Once I introduce windowing even with aggressive watermark it seems it takes
at least a second (often multiple seconds) to see a window fire.

Same appears true for setting events timers in the State API; there seems a
delay between setting a timer at current event time and the timer callback
fires.

Are there any good/canonical latency benchmarks/reports for different
runners?

My next move may be to evaluate Flink in terms of these latencies, but
curious if I should be trying to get sub-second latencies out of Beam
(windowing) at all?

Re: Sub-second Beam

[Aaron Dixon <at...@gmail.com>]

Ah great point about the Kafka partition skew. I can see how the problem
domain is kind of stacked against sub-second latency expectations. Thanks
again.

On Fri, Dec 13, 2019 at 3:48 PM Robert Bradshaw <ro...@google.com> wrote:

> On Fri, Dec 13, 2019 at 1:20 PM Aaron Dixon <at...@gmail.com> wrote:
> >
> > Thank you Robert, this is really helpful context and confirmation.
> >
> > > You mention having an "agressive" watermark--could you clarify what
> you mean by this?
> >
> > I'm using KafkaIO and I customize the watermark to follow my message
> event timestamps as they come (instead of having it lag behind the stream
> to allow for any even slightly out-of-order events.)  So I call this
> "aggressive".
>
> What about skew between partitions? This would show up as a difference
> between the oldest and newest element being concurrently processed,
> and would hold back the watermark.
>
> > I also have a custom window that closes precisely on certain event
> markers in the stream -- and I need these windows to trigger as fast as
> possible (was hoping for sub-second here, but I suspected that was not the
> focus of runners as you're saying so I think I can live w/o this).
>
> Correct.
>
> Essentially, state and window-using operations are aggregating
> operations, and the latency is bound by how simultaneously every set
> of elements that need to be aggregated arrive together. This kind of
> synchronization is difficult to do in a distributed system, and
> watermarks are an attempt to accurately track the minimum time needed
> to wait to ensure all events have arrived and a result can be emitted.
>
> > > Certainly all-in-process runners would likely take the cake here
> >
> > Besides DirectRunner which as I understand it is primarily for testing,
> what are other in-process runners?
>
> Flink and Spark have (test) all-in-process modes too.
>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Dec 13, 2019 at 1:45 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>
> >> In general, sub-second latencies are difficult because one must wait
> >> for the watermark to catch up before actually firing. This would
> >> require the oldest item in flight across all machines to be almost
> >> exactly the same timestamp as the newest. Furthermore most sources
> >> cannot provide sub-second watermarks. You mention having an
> >> "agressive" watermark--could you clarify what you mean by this? There
> >> are also (generally small) latencies involved with persisting state
> >> and then firing timers, and windowing is built on the same mechanisms.
> >>
> >> I am not aware of latency benchmarks for various runners--in my
> >> experience most people are interested in high throughput at
> >> O(second-minute) latency. There's nothing in the Beam model that
> >> prevents sub-second latencies, but I don't think this has been pushed
> >> very far on most runners. Gathering such data would certainly be
> >> interesting. (Certainly all-in-process runners would likely take the
> >> cake here, but it'd be interesting to see how it degrades as one adds
> >> more machines.)
> >>
> >> On Fri, Dec 13, 2019 at 10:58 AM Aaron Dixon <at...@gmail.com> wrote:
> >> >
> >> > I've been building pipelines and benchmarking Beam jobs in Dataflow.
> >> >
> >> > Without windowing, latencies look pretty good (reliably sub-second)
> from ingest to sink.
> >> >
> >> > Once I introduce windowing even with aggressive watermark it seems it
> takes at least a second (often multiple seconds) to see a window fire.
> >> >
> >> > Same appears true for setting events timers in the State API; there
> seems a delay between setting a timer at current event time and the timer
> callback fires.
> >> >
> >> > Are there any good/canonical latency benchmarks/reports for different
> runners?
> >> >
> >> > My next move may be to evaluate Flink in terms of these latencies,
> but curious if I should be trying to get sub-second latencies out of Beam
> (windowing) at all?
> >> >
>

Re: Sub-second Beam

[Aaron Dixon <at...@gmail.com>]

Mehran,

My benchmarking is very ad hoc and I don't have the resources at the moment
to do something formal or publish anything (but I may in the coming year.)
The short summary from cursory online research, the feedback in this email
thread, and my own empirical evidence (even taking KafkaIO out of the
equation and just feeding in a manual data set via Beam's `Create.of()`), I
am generally seeing 1 - 5 seconds in Dataflow of delay after I expect the
watermark to break the end of my window. This is very limited testing and
I'm using very small hand-tailored code w/ custom windowing; my main take
home has been that I shouldn't expect sub-second latencies at all and that
I'm more inclined to see multiple single-digit seconds.

This is fairly helpful StackOverflow thread that looks like they've done
more due diligence than I have:
https://stackoverflow.com/questions/39571154/google-cloud-dataflow-latency-for-real-time-processing
And
they come up with 1.5-1.7s as best case.



On Fri, Dec 13, 2019 at 4:46 PM Mehran Nazir <mn...@google.com> wrote:

> Hey Aaron,
>
> Do you plan on publishing the results of your test, as well as a
> description of your testing method? I'd be interested in seeing what you
> find.
>
> Mehran
>
> On Fri, Dec 13, 2019 at 4:48 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> On Fri, Dec 13, 2019 at 1:20 PM Aaron Dixon <at...@gmail.com> wrote:
>> >
>> > Thank you Robert, this is really helpful context and confirmation.
>> >
>> > > You mention having an "agressive" watermark--could you clarify what
>> you mean by this?
>> >
>> > I'm using KafkaIO and I customize the watermark to follow my message
>> event timestamps as they come (instead of having it lag behind the stream
>> to allow for any even slightly out-of-order events.)  So I call this
>> "aggressive".
>>
>> What about skew between partitions? This would show up as a difference
>> between the oldest and newest element being concurrently processed,
>> and would hold back the watermark.
>>
>> > I also have a custom window that closes precisely on certain event
>> markers in the stream -- and I need these windows to trigger as fast as
>> possible (was hoping for sub-second here, but I suspected that was not the
>> focus of runners as you're saying so I think I can live w/o this).
>>
>> Correct.
>>
>> Essentially, state and window-using operations are aggregating
>> operations, and the latency is bound by how simultaneously every set
>> of elements that need to be aggregated arrive together. This kind of
>> synchronization is difficult to do in a distributed system, and
>> watermarks are an attempt to accurately track the minimum time needed
>> to wait to ensure all events have arrived and a result can be emitted.
>>
>> > > Certainly all-in-process runners would likely take the cake here
>> >
>> > Besides DirectRunner which as I understand it is primarily for testing,
>> what are other in-process runners?
>>
>> Flink and Spark have (test) all-in-process modes too.
>>
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Dec 13, 2019 at 1:45 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>
>> >> In general, sub-second latencies are difficult because one must wait
>> >> for the watermark to catch up before actually firing. This would
>> >> require the oldest item in flight across all machines to be almost
>> >> exactly the same timestamp as the newest. Furthermore most sources
>> >> cannot provide sub-second watermarks. You mention having an
>> >> "agressive" watermark--could you clarify what you mean by this? There
>> >> are also (generally small) latencies involved with persisting state
>> >> and then firing timers, and windowing is built on the same mechanisms.
>> >>
>> >> I am not aware of latency benchmarks for various runners--in my
>> >> experience most people are interested in high throughput at
>> >> O(second-minute) latency. There's nothing in the Beam model that
>> >> prevents sub-second latencies, but I don't think this has been pushed
>> >> very far on most runners. Gathering such data would certainly be
>> >> interesting. (Certainly all-in-process runners would likely take the
>> >> cake here, but it'd be interesting to see how it degrades as one adds
>> >> more machines.)
>> >>
>> >> On Fri, Dec 13, 2019 at 10:58 AM Aaron Dixon <at...@gmail.com>
>> wrote:
>> >> >
>> >> > I've been building pipelines and benchmarking Beam jobs in Dataflow.
>> >> >
>> >> > Without windowing, latencies look pretty good (reliably sub-second)
>> from ingest to sink.
>> >> >
>> >> > Once I introduce windowing even with aggressive watermark it seems
>> it takes at least a second (often multiple seconds) to see a window fire.
>> >> >
>> >> > Same appears true for setting events timers in the State API; there
>> seems a delay between setting a timer at current event time and the timer
>> callback fires.
>> >> >
>> >> > Are there any good/canonical latency benchmarks/reports for
>> different runners?
>> >> >
>> >> > My next move may be to evaluate Flink in terms of these latencies,
>> but curious if I should be trying to get sub-second latencies out of Beam
>> (windowing) at all?
>> >> >
>>
>
>
> --
>
> Mehran Nazir
>
> mnazir@google.com
>
> Product Manager, Dataflow
>

Re: Sub-second Beam

[Mehran Nazir <mn...@google.com>]

Hey Aaron,

Do you plan on publishing the results of your test, as well as a
description of your testing method? I'd be interested in seeing what you
find.

Mehran

On Fri, Dec 13, 2019 at 4:48 PM Robert Bradshaw <ro...@google.com> wrote:

> On Fri, Dec 13, 2019 at 1:20 PM Aaron Dixon <at...@gmail.com> wrote:
> >
> > Thank you Robert, this is really helpful context and confirmation.
> >
> > > You mention having an "agressive" watermark--could you clarify what
> you mean by this?
> >
> > I'm using KafkaIO and I customize the watermark to follow my message
> event timestamps as they come (instead of having it lag behind the stream
> to allow for any even slightly out-of-order events.)  So I call this
> "aggressive".
>
> What about skew between partitions? This would show up as a difference
> between the oldest and newest element being concurrently processed,
> and would hold back the watermark.
>
> > I also have a custom window that closes precisely on certain event
> markers in the stream -- and I need these windows to trigger as fast as
> possible (was hoping for sub-second here, but I suspected that was not the
> focus of runners as you're saying so I think I can live w/o this).
>
> Correct.
>
> Essentially, state and window-using operations are aggregating
> operations, and the latency is bound by how simultaneously every set
> of elements that need to be aggregated arrive together. This kind of
> synchronization is difficult to do in a distributed system, and
> watermarks are an attempt to accurately track the minimum time needed
> to wait to ensure all events have arrived and a result can be emitted.
>
> > > Certainly all-in-process runners would likely take the cake here
> >
> > Besides DirectRunner which as I understand it is primarily for testing,
> what are other in-process runners?
>
> Flink and Spark have (test) all-in-process modes too.
>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Dec 13, 2019 at 1:45 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>
> >> In general, sub-second latencies are difficult because one must wait
> >> for the watermark to catch up before actually firing. This would
> >> require the oldest item in flight across all machines to be almost
> >> exactly the same timestamp as the newest. Furthermore most sources
> >> cannot provide sub-second watermarks. You mention having an
> >> "agressive" watermark--could you clarify what you mean by this? There
> >> are also (generally small) latencies involved with persisting state
> >> and then firing timers, and windowing is built on the same mechanisms.
> >>
> >> I am not aware of latency benchmarks for various runners--in my
> >> experience most people are interested in high throughput at
> >> O(second-minute) latency. There's nothing in the Beam model that
> >> prevents sub-second latencies, but I don't think this has been pushed
> >> very far on most runners. Gathering such data would certainly be
> >> interesting. (Certainly all-in-process runners would likely take the
> >> cake here, but it'd be interesting to see how it degrades as one adds
> >> more machines.)
> >>
> >> On Fri, Dec 13, 2019 at 10:58 AM Aaron Dixon <at...@gmail.com> wrote:
> >> >
> >> > I've been building pipelines and benchmarking Beam jobs in Dataflow.
> >> >
> >> > Without windowing, latencies look pretty good (reliably sub-second)
> from ingest to sink.
> >> >
> >> > Once I introduce windowing even with aggressive watermark it seems it
> takes at least a second (often multiple seconds) to see a window fire.
> >> >
> >> > Same appears true for setting events timers in the State API; there
> seems a delay between setting a timer at current event time and the timer
> callback fires.
> >> >
> >> > Are there any good/canonical latency benchmarks/reports for different
> runners?
> >> >
> >> > My next move may be to evaluate Flink in terms of these latencies,
> but curious if I should be trying to get sub-second latencies out of Beam
> (windowing) at all?
> >> >
>


-- 

Mehran Nazir

mnazir@google.com

Product Manager, Dataflow

Re: Sub-second Beam

[Robert Bradshaw <ro...@google.com>]

On Fri, Dec 13, 2019 at 1:20 PM Aaron Dixon <at...@gmail.com> wrote:
>
> Thank you Robert, this is really helpful context and confirmation.
>
> > You mention having an "agressive" watermark--could you clarify what you mean by this?
>
> I'm using KafkaIO and I customize the watermark to follow my message event timestamps as they come (instead of having it lag behind the stream to allow for any even slightly out-of-order events.)  So I call this "aggressive".

What about skew between partitions? This would show up as a difference
between the oldest and newest element being concurrently processed,
and would hold back the watermark.

> I also have a custom window that closes precisely on certain event markers in the stream -- and I need these windows to trigger as fast as possible (was hoping for sub-second here, but I suspected that was not the focus of runners as you're saying so I think I can live w/o this).

Correct.

Essentially, state and window-using operations are aggregating
operations, and the latency is bound by how simultaneously every set
of elements that need to be aggregated arrive together. This kind of
synchronization is difficult to do in a distributed system, and
watermarks are an attempt to accurately track the minimum time needed
to wait to ensure all events have arrived and a result can be emitted.

> > Certainly all-in-process runners would likely take the cake here
>
> Besides DirectRunner which as I understand it is primarily for testing, what are other in-process runners?

Flink and Spark have (test) all-in-process modes too.

>
>
>
>
>
>
>
>
>
>
> On Fri, Dec 13, 2019 at 1:45 PM Robert Bradshaw <ro...@google.com> wrote:
>>
>> In general, sub-second latencies are difficult because one must wait
>> for the watermark to catch up before actually firing. This would
>> require the oldest item in flight across all machines to be almost
>> exactly the same timestamp as the newest. Furthermore most sources
>> cannot provide sub-second watermarks. You mention having an
>> "agressive" watermark--could you clarify what you mean by this? There
>> are also (generally small) latencies involved with persisting state
>> and then firing timers, and windowing is built on the same mechanisms.
>>
>> I am not aware of latency benchmarks for various runners--in my
>> experience most people are interested in high throughput at
>> O(second-minute) latency. There's nothing in the Beam model that
>> prevents sub-second latencies, but I don't think this has been pushed
>> very far on most runners. Gathering such data would certainly be
>> interesting. (Certainly all-in-process runners would likely take the
>> cake here, but it'd be interesting to see how it degrades as one adds
>> more machines.)
>>
>> On Fri, Dec 13, 2019 at 10:58 AM Aaron Dixon <at...@gmail.com> wrote:
>> >
>> > I've been building pipelines and benchmarking Beam jobs in Dataflow.
>> >
>> > Without windowing, latencies look pretty good (reliably sub-second) from ingest to sink.
>> >
>> > Once I introduce windowing even with aggressive watermark it seems it takes at least a second (often multiple seconds) to see a window fire.
>> >
>> > Same appears true for setting events timers in the State API; there seems a delay between setting a timer at current event time and the timer callback fires.
>> >
>> > Are there any good/canonical latency benchmarks/reports for different runners?
>> >
>> > My next move may be to evaluate Flink in terms of these latencies, but curious if I should be trying to get sub-second latencies out of Beam (windowing) at all?
>> >

Re: Sub-second Beam

[Aaron Dixon <at...@gmail.com>]

Thank you Robert, this is really helpful context and confirmation.

> You mention having an "agressive" watermark--could you clarify what you
mean by this?

I'm using KafkaIO and I customize the watermark to follow my message event
timestamps as they come (instead of having it lag behind the stream to
allow for any even slightly out-of-order events.) So I call this
"aggressive". I also have a custom window that closes precisely on certain
event markers in the stream -- and I need these windows to trigger as fast
as possible (was hoping for sub-second here, but I suspected that was not
the focus of runners as you're saying so I think I can live w/o this).

> Certainly all-in-process runners would likely take the cake here

Besides DirectRunner which as I understand it is primarily for testing,
what are other in-process runners?










On Fri, Dec 13, 2019 at 1:45 PM Robert Bradshaw <ro...@google.com> wrote:

> In general, sub-second latencies are difficult because one must wait
> for the watermark to catch up before actually firing. This would
> require the oldest item in flight across all machines to be almost
> exactly the same timestamp as the newest. Furthermore most sources
> cannot provide sub-second watermarks. You mention having an
> "agressive" watermark--could you clarify what you mean by this? There
> are also (generally small) latencies involved with persisting state
> and then firing timers, and windowing is built on the same mechanisms.
>
> I am not aware of latency benchmarks for various runners--in my
> experience most people are interested in high throughput at
> O(second-minute) latency. There's nothing in the Beam model that
> prevents sub-second latencies, but I don't think this has been pushed
> very far on most runners. Gathering such data would certainly be
> interesting. (Certainly all-in-process runners would likely take the
> cake here, but it'd be interesting to see how it degrades as one adds
> more machines.)
>
> On Fri, Dec 13, 2019 at 10:58 AM Aaron Dixon <at...@gmail.com> wrote:
> >
> > I've been building pipelines and benchmarking Beam jobs in Dataflow.
> >
> > Without windowing, latencies look pretty good (reliably sub-second) from
> ingest to sink.
> >
> > Once I introduce windowing even with aggressive watermark it seems it
> takes at least a second (often multiple seconds) to see a window fire.
> >
> > Same appears true for setting events timers in the State API; there
> seems a delay between setting a timer at current event time and the timer
> callback fires.
> >
> > Are there any good/canonical latency benchmarks/reports for different
> runners?
> >
> > My next move may be to evaluate Flink in terms of these latencies, but
> curious if I should be trying to get sub-second latencies out of Beam
> (windowing) at all?
> >
>

Re: Sub-second Beam

[Robert Bradshaw <ro...@google.com>]

In general, sub-second latencies are difficult because one must wait
for the watermark to catch up before actually firing. This would
require the oldest item in flight across all machines to be almost
exactly the same timestamp as the newest. Furthermore most sources
cannot provide sub-second watermarks. You mention having an
"agressive" watermark--could you clarify what you mean by this? There
are also (generally small) latencies involved with persisting state
and then firing timers, and windowing is built on the same mechanisms.

I am not aware of latency benchmarks for various runners--in my
experience most people are interested in high throughput at
O(second-minute) latency. There's nothing in the Beam model that
prevents sub-second latencies, but I don't think this has been pushed
very far on most runners. Gathering such data would certainly be
interesting. (Certainly all-in-process runners would likely take the
cake here, but it'd be interesting to see how it degrades as one adds
more machines.)

On Fri, Dec 13, 2019 at 10:58 AM Aaron Dixon <at...@gmail.com> wrote:
>
> I've been building pipelines and benchmarking Beam jobs in Dataflow.
>
> Without windowing, latencies look pretty good (reliably sub-second) from ingest to sink.
>
> Once I introduce windowing even with aggressive watermark it seems it takes at least a second (often multiple seconds) to see a window fire.
>
> Same appears true for setting events timers in the State API; there seems a delay between setting a timer at current event time and the timer callback fires.
>
> Are there any good/canonical latency benchmarks/reports for different runners?
>
> My next move may be to evaluate Flink in terms of these latencies, but curious if I should be trying to get sub-second latencies out of Beam (windowing) at all?
>